2020-10-29T07:53:28.603 INFO:root:teuthology version: 1.0.0-f7de558 2020-10-29T07:53:28.605 INFO:teuthology.results:subset: '0/10' 2020-10-29T07:53:28.606 INFO:teuthology.results:seed: '1804' 2020-10-29T11:27:11.662 INFO:root:teuthology version: 1.0.0-f7de558 2020-10-29T11:27:12.037 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-10-29T22:43:37.039 INFO:teuthology.results:Tests finished! gathering results... 2020-10-29T22:44:40.731 INFO:scrape:Found 389 jobs 2020-10-29T22:44:41.269 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6268-g944bb62c (944bb62c829d723d3f38fa6a8966ac655e3866f2) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fab373e09f6] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x275c10) [0x7fab373e0c10] 2020-10-29T22:44:41.271 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:08.892 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2020-10-29T08:38:08.892 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2020-10-29T08:38:08.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2020-10-29T22:44:41.272 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.242 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", , 2020-10-29T08:38:09.242 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:35:06.450111+0000", , 2020-10-29T08:38:09.242 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0038871159999999999 2020-10-29T22:44:41.272 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.260 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", , 2020-10-29T08:38:09.260 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:35:06.450139+0000", , 2020-10-29T08:38:09.261 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6612000000000004e-05 2020-10-29T22:44:41.273 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.276 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for missing object", , 2020-10-29T08:38:09.276 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:35:06.440066+0000", , 2020-10-29T08:38:09.276 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010209104 2020-10-29T22:44:41.273 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.290 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", , 2020-10-29T08:38:09.290 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:35:06.440117+0000", , 2020-10-29T08:38:09.290 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9179999999999998e-05 2020-10-29T22:44:41.274 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.304 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", , 2020-10-29T08:38:09.304 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:35:05.328002+0000", , 2020-10-29T08:38:09.305 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.112248436 2020-10-29T22:44:41.275 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.319 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", , 2020-10-29T08:38:09.319 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:35:05.328124+0000", , 2020-10-29T08:38:09.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.21214798800000001 2020-10-29T22:44:41.275 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.333 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", , 2020-10-29T08:38:09.333 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:35:05.548777+0000", , 2020-10-29T08:38:09.333 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.22072412799999999 2020-10-29T22:44:41.276 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.346 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", , 2020-10-29T08:38:09.346 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:35:06.461085+0000", , 2020-10-29T08:38:09.347 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00064210000000000005 2020-10-29T22:44:41.276 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.360 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", , 2020-10-29T08:38:09.360 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:35:06.452291+0000", , 2020-10-29T08:38:09.361 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0095064469999999995 2020-10-29T22:44:41.278 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.375 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 122.99377662800001, , 2020-10-29T08:38:09.375 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.036163629000000003, , 2020-10-29T08:38:09.375 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2020-10-29T22:44:41.278 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.389 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2020-10-29T08:38:09.389 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", , 2020-10-29T08:38:09.389 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:36:06.233807+0000", 2020-10-29T22:44:41.279 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.403 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2020-10-29T08:38:09.403 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_committed", , 2020-10-29T08:38:09.403 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:36:44.308920+0000", 2020-10-29T22:44:41.279 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.417 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2020-10-29T08:36:58.412599+0000", , 2020-10-29T08:38:09.417 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 70.814814772000005, , 2020-10-29T08:38:09.417 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.037042330999999998, 2020-10-29T22:44:41.280 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.431 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2020-10-29T08:38:09.431 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2020-10-29T08:38:09.431 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2020-10-29T22:44:41.281 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2020-10-29T08:38:09.445 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2020-10-29T08:38:09.445 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2020-10-29T22:44:41.281 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.459 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6221000000000003e-05 , 2020-10-29T08:38:09.459 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2020-10-29T08:38:09.459 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-10-29T22:44:41.282 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.472 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0649999999999998e-06 , 2020-10-29T08:38:09.473 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2020-10-29T08:38:09.473 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-10-29T22:44:41.282 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.616 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2020-10-29T08:38:09.616 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2020-10-29T08:38:09.616 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2020-10-29T22:44:41.283 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.632 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2020-10-29T08:38:09.632 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2020-10-29T08:38:09.632 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2020-10-29T22:44:41.290 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status , raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' 2020-10-29T22:44:41.292 WARNING:scrape:Ignoring malparsed backtrace: mon.a: mon_osd_down_out_interval = '600' (not observed, change may require restart) , mon.b: mon_osd_down_out_interval = '600' (not observed, change may require restart) , mon.c: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2020-10-29T22:44:41.294 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:39:01.231 DEBUG:teuthology.run_tasks:Unwinding manager ceph , 2020-10-29T08:39:01.241 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... , 2020-10-29T08:39:01.241 INFO:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-10-29T22:44:41.296 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:39:08.300 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2020-10-29T08:38:21.748941+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=38, tm_sec=21, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=39, tm_sec=2, tm_wday=3, tm_yday=303, tm_isdst=0) , 2020-10-29T08:39:08.301 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2020-10-29T08:38:44.110183+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=38, tm_sec=44, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=39, tm_sec=2, tm_wday=3, tm_yday=303, tm_isdst=0) , 2020-10-29T08:39:08.301 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2020-10-29T08:38:36.373276+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=38, tm_sec=36, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=39, tm_sec=2, tm_wday=3, tm_yday=303, tm_isdst=0) 2020-10-29T22:44:41.298 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:39:28.699 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2020-10-29T08:38:26.886555+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=38, tm_sec=26, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=39, tm_sec=2, tm_wday=3, tm_yday=303, tm_isdst=0) , 2020-10-29T08:39:28.699 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2020-10-29T08:38:40.597503+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=38, tm_sec=40, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=39, tm_sec=2, tm_wday=3, tm_yday=303, tm_isdst=0) , 2020-10-29T08:39:28.700 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2020-10-29T08:38:42.184522+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=38, tm_sec=42, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=39, tm_sec=2, tm_wday=3, tm_yday=303, tm_isdst=0) 2020-10-29T22:44:41.299 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:40:30.673 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-29T08:40:30.683 INFO:tasks.ceph.mon.b:Stopped , 2020-10-29T08:40:30.683 INFO:tasks.ceph:Checking cluster log for badness... 2020-10-29T22:44:41.299 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:41:02.865 INFO:teuthology.orchestra.run.smithi143.stdout:^* hv02.front.sepia.ceph.com 2 6 377 56 +625us[ +627us] +/- 48ms , 2020-10-29T08:41:02.865 INFO:teuthology.orchestra.run.smithi143.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 55 -84us[ -84us] +/- 70ms , 2020-10-29T08:41:02.865 INFO:teuthology.orchestra.run.smithi143.stdout:^+ hv04.front.sepia.ceph.com 3 6 377 55 -745us[ -745us] +/- 40ms 2020-10-29T22:44:41.300 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 334, in fetch_binaries_for_coredumps , remote.get_file(debug_path, coredump_path) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 405, in get_file 2020-10-29T22:44:41.512 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6268-g944bb62c (944bb62c829d723d3f38fa6a8966ac655e3866f2) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fab373e09f6] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x275c10) [0x7fab373e0c10] 2020-10-29T22:44:41.513 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:08.892 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2020-10-29T08:38:08.892 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2020-10-29T08:38:08.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2020-10-29T22:44:41.514 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.242 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", , 2020-10-29T08:38:09.242 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:35:06.450111+0000", , 2020-10-29T08:38:09.242 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0038871159999999999 2020-10-29T22:44:41.514 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.260 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", , 2020-10-29T08:38:09.260 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:35:06.450139+0000", , 2020-10-29T08:38:09.261 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6612000000000004e-05 2020-10-29T22:44:41.515 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.276 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for missing object", , 2020-10-29T08:38:09.276 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:35:06.440066+0000", , 2020-10-29T08:38:09.276 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010209104 2020-10-29T22:44:41.515 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.290 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", , 2020-10-29T08:38:09.290 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:35:06.440117+0000", , 2020-10-29T08:38:09.290 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9179999999999998e-05 2020-10-29T22:44:41.516 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.304 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", , 2020-10-29T08:38:09.304 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:35:05.328002+0000", , 2020-10-29T08:38:09.305 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.112248436 2020-10-29T22:44:41.516 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.319 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", , 2020-10-29T08:38:09.319 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:35:05.328124+0000", , 2020-10-29T08:38:09.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.21214798800000001 2020-10-29T22:44:41.517 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.333 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", , 2020-10-29T08:38:09.333 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:35:05.548777+0000", , 2020-10-29T08:38:09.333 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.22072412799999999 2020-10-29T22:44:41.517 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.346 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", , 2020-10-29T08:38:09.346 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:35:06.461085+0000", , 2020-10-29T08:38:09.347 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00064210000000000005 2020-10-29T22:44:41.518 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.360 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", , 2020-10-29T08:38:09.360 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:35:06.452291+0000", , 2020-10-29T08:38:09.361 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0095064469999999995 2020-10-29T22:44:41.518 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.375 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 122.99377662800001, , 2020-10-29T08:38:09.375 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.036163629000000003, , 2020-10-29T08:38:09.375 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2020-10-29T22:44:41.519 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.389 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2020-10-29T08:38:09.389 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", , 2020-10-29T08:38:09.389 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:36:06.233807+0000", 2020-10-29T22:44:41.519 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.403 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2020-10-29T08:38:09.403 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_committed", , 2020-10-29T08:38:09.403 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:36:44.308920+0000", 2020-10-29T22:44:41.520 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.417 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2020-10-29T08:36:58.412599+0000", , 2020-10-29T08:38:09.417 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 70.814814772000005, , 2020-10-29T08:38:09.417 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.037042330999999998, 2020-10-29T22:44:41.520 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.431 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2020-10-29T08:38:09.431 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2020-10-29T08:38:09.431 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2020-10-29T22:44:41.521 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2020-10-29T08:38:09.445 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2020-10-29T08:38:09.445 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2020-10-29T22:44:41.521 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.459 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6221000000000003e-05 , 2020-10-29T08:38:09.459 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2020-10-29T08:38:09.459 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-10-29T22:44:41.521 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.472 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0649999999999998e-06 , 2020-10-29T08:38:09.473 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2020-10-29T08:38:09.473 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-10-29T22:44:41.522 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.616 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2020-10-29T08:38:09.616 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2020-10-29T08:38:09.616 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2020-10-29T22:44:41.523 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.632 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2020-10-29T08:38:09.632 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2020-10-29T08:38:09.632 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2020-10-29T22:44:41.529 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status , raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' 2020-10-29T22:44:41.530 WARNING:scrape:Ignoring malparsed backtrace: mon.a: mon_osd_down_out_interval = '600' (not observed, change may require restart) , mon.b: mon_osd_down_out_interval = '600' (not observed, change may require restart) , mon.c: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2020-10-29T22:44:41.532 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:39:01.231 DEBUG:teuthology.run_tasks:Unwinding manager ceph , 2020-10-29T08:39:01.241 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... , 2020-10-29T08:39:01.241 INFO:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-10-29T22:44:41.534 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:39:08.300 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2020-10-29T08:38:21.748941+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=38, tm_sec=21, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=39, tm_sec=2, tm_wday=3, tm_yday=303, tm_isdst=0) , 2020-10-29T08:39:08.301 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2020-10-29T08:38:44.110183+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=38, tm_sec=44, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=39, tm_sec=2, tm_wday=3, tm_yday=303, tm_isdst=0) , 2020-10-29T08:39:08.301 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2020-10-29T08:38:36.373276+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=38, tm_sec=36, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=39, tm_sec=2, tm_wday=3, tm_yday=303, tm_isdst=0) 2020-10-29T22:44:41.536 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:39:28.699 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2020-10-29T08:38:26.886555+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=38, tm_sec=26, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=39, tm_sec=2, tm_wday=3, tm_yday=303, tm_isdst=0) , 2020-10-29T08:39:28.699 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2020-10-29T08:38:40.597503+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=38, tm_sec=40, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=39, tm_sec=2, tm_wday=3, tm_yday=303, tm_isdst=0) , 2020-10-29T08:39:28.700 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2020-10-29T08:38:42.184522+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=38, tm_sec=42, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=39, tm_sec=2, tm_wday=3, tm_yday=303, tm_isdst=0) 2020-10-29T22:44:41.537 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:40:30.673 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-29T08:40:30.683 INFO:tasks.ceph.mon.b:Stopped , 2020-10-29T08:40:30.683 INFO:tasks.ceph:Checking cluster log for badness... 2020-10-29T22:44:41.537 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:41:02.865 INFO:teuthology.orchestra.run.smithi143.stdout:^* hv02.front.sepia.ceph.com 2 6 377 56 +625us[ +627us] +/- 48ms , 2020-10-29T08:41:02.865 INFO:teuthology.orchestra.run.smithi143.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 55 -84us[ -84us] +/- 70ms , 2020-10-29T08:41:02.865 INFO:teuthology.orchestra.run.smithi143.stdout:^+ hv04.front.sepia.ceph.com 3 6 377 55 -745us[ -745us] +/- 40ms 2020-10-29T22:44:41.538 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 334, in fetch_binaries_for_coredumps , remote.get_file(debug_path, coredump_path) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 405, in get_file 2020-10-29T22:44:41.758 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6268-g944bb62c (944bb62c829d723d3f38fa6a8966ac655e3866f2) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fab373e09f6] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x275c10) [0x7fab373e0c10] 2020-10-29T22:44:41.759 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:08.892 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2020-10-29T08:38:08.892 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2020-10-29T08:38:08.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2020-10-29T22:44:41.760 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.242 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", , 2020-10-29T08:38:09.242 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:35:06.450111+0000", , 2020-10-29T08:38:09.242 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0038871159999999999 2020-10-29T22:44:41.760 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.260 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", , 2020-10-29T08:38:09.260 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:35:06.450139+0000", , 2020-10-29T08:38:09.261 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6612000000000004e-05 2020-10-29T22:44:41.760 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.276 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for missing object", , 2020-10-29T08:38:09.276 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:35:06.440066+0000", , 2020-10-29T08:38:09.276 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010209104 2020-10-29T22:44:41.761 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.290 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", , 2020-10-29T08:38:09.290 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:35:06.440117+0000", , 2020-10-29T08:38:09.290 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9179999999999998e-05 2020-10-29T22:44:41.761 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.304 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", , 2020-10-29T08:38:09.304 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:35:05.328002+0000", , 2020-10-29T08:38:09.305 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.112248436 2020-10-29T22:44:41.762 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.319 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", , 2020-10-29T08:38:09.319 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:35:05.328124+0000", , 2020-10-29T08:38:09.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.21214798800000001 2020-10-29T22:44:41.762 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.333 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", , 2020-10-29T08:38:09.333 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:35:05.548777+0000", , 2020-10-29T08:38:09.333 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.22072412799999999 2020-10-29T22:44:41.763 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.346 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", , 2020-10-29T08:38:09.346 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:35:06.461085+0000", , 2020-10-29T08:38:09.347 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00064210000000000005 2020-10-29T22:44:41.763 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.360 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", , 2020-10-29T08:38:09.360 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:35:06.452291+0000", , 2020-10-29T08:38:09.361 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0095064469999999995 2020-10-29T22:44:41.764 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.375 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 122.99377662800001, , 2020-10-29T08:38:09.375 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.036163629000000003, , 2020-10-29T08:38:09.375 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2020-10-29T22:44:41.764 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.389 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2020-10-29T08:38:09.389 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", , 2020-10-29T08:38:09.389 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:36:06.233807+0000", 2020-10-29T22:44:41.765 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.403 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2020-10-29T08:38:09.403 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_committed", , 2020-10-29T08:38:09.403 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2020-10-29T08:36:44.308920+0000", 2020-10-29T22:44:41.765 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.417 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2020-10-29T08:36:58.412599+0000", , 2020-10-29T08:38:09.417 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 70.814814772000005, , 2020-10-29T08:38:09.417 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.037042330999999998, 2020-10-29T22:44:41.766 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.431 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2020-10-29T08:38:09.431 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2020-10-29T08:38:09.431 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2020-10-29T22:44:41.766 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2020-10-29T08:38:09.445 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2020-10-29T08:38:09.445 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2020-10-29T22:44:41.767 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.459 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6221000000000003e-05 , 2020-10-29T08:38:09.459 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2020-10-29T08:38:09.459 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-10-29T22:44:41.767 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.472 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0649999999999998e-06 , 2020-10-29T08:38:09.473 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2020-10-29T08:38:09.473 INFO:teuthology.orchestra.run.smithi154.stdout: { 2020-10-29T22:44:41.768 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.616 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2020-10-29T08:38:09.616 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2020-10-29T08:38:09.616 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2020-10-29T22:44:41.768 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:38:09.632 INFO:teuthology.orchestra.run.smithi154.stdout: }, , 2020-10-29T08:38:09.632 INFO:teuthology.orchestra.run.smithi154.stdout: { , 2020-10-29T08:38:09.632 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2020-10-29T22:44:41.775 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status , raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' 2020-10-29T22:44:41.777 WARNING:scrape:Ignoring malparsed backtrace: mon.a: mon_osd_down_out_interval = '600' (not observed, change may require restart) , mon.b: mon_osd_down_out_interval = '600' (not observed, change may require restart) , mon.c: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2020-10-29T22:44:41.779 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:39:01.231 DEBUG:teuthology.run_tasks:Unwinding manager ceph , 2020-10-29T08:39:01.241 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... , 2020-10-29T08:39:01.241 INFO:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json 2020-10-29T22:44:41.780 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:39:08.300 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2020-10-29T08:38:21.748941+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=38, tm_sec=21, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=39, tm_sec=2, tm_wday=3, tm_yday=303, tm_isdst=0) , 2020-10-29T08:39:08.301 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2020-10-29T08:38:44.110183+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=38, tm_sec=44, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=39, tm_sec=2, tm_wday=3, tm_yday=303, tm_isdst=0) , 2020-10-29T08:39:08.301 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2020-10-29T08:38:36.373276+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=38, tm_sec=36, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=39, tm_sec=2, tm_wday=3, tm_yday=303, tm_isdst=0) 2020-10-29T22:44:41.783 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:39:28.699 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2020-10-29T08:38:26.886555+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=38, tm_sec=26, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=39, tm_sec=2, tm_wday=3, tm_yday=303, tm_isdst=0) , 2020-10-29T08:39:28.699 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2020-10-29T08:38:40.597503+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=38, tm_sec=40, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=39, tm_sec=2, tm_wday=3, tm_yday=303, tm_isdst=0) , 2020-10-29T08:39:28.700 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2020-10-29T08:38:42.184522+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=38, tm_sec=42, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=8, tm_min=39, tm_sec=2, tm_wday=3, tm_yday=303, tm_isdst=0) 2020-10-29T22:44:41.784 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:40:30.673 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-29T08:40:30.683 INFO:tasks.ceph.mon.b:Stopped , 2020-10-29T08:40:30.683 INFO:tasks.ceph:Checking cluster log for badness... 2020-10-29T22:44:41.784 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:41:02.865 INFO:teuthology.orchestra.run.smithi143.stdout:^* hv02.front.sepia.ceph.com 2 6 377 56 +625us[ +627us] +/- 48ms , 2020-10-29T08:41:02.865 INFO:teuthology.orchestra.run.smithi143.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 55 -84us[ -84us] +/- 70ms , 2020-10-29T08:41:02.865 INFO:teuthology.orchestra.run.smithi143.stdout:^+ hv04.front.sepia.ceph.com 3 6 377 55 -745us[ -745us] +/- 40ms 2020-10-29T22:44:41.785 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 334, in fetch_binaries_for_coredumps , remote.get_file(debug_path, coredump_path) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 405, in get_file 2020-10-29T22:44:42.208 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6268-g944bb62c (944bb62c829d723d3f38fa6a8966ac655e3866f2) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7fc8814e0cfb] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x55aedc0869b0] 2020-10-29T22:44:42.209 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:47.930 INFO:teuthology.orchestra.run.smithi036.stdout: "duration": 8.9020000000000005e-06 , 2020-10-29T09:30:47.930 INFO:teuthology.orchestra.run.smithi036.stdout: }, , 2020-10-29T09:30:47.930 INFO:teuthology.orchestra.run.smithi036.stdout: { 2020-10-29T22:44:42.213 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:47.946 INFO:teuthology.orchestra.run.smithi036.stdout: "duration": 0.0091294589999999995 , 2020-10-29T09:30:47.946 INFO:teuthology.orchestra.run.smithi036.stdout: }, , 2020-10-29T09:30:47.946 INFO:teuthology.orchestra.run.smithi036.stdout: { 2020-10-29T22:44:42.214 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:47.961 INFO:teuthology.orchestra.run.smithi036.stdout: "time": "2020-10-29T09:28:01.262386+0000", , 2020-10-29T09:30:47.961 INFO:teuthology.orchestra.run.smithi036.stdout: "duration": 0.000113807 , 2020-10-29T09:30:47.961 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2020-10-29T22:44:42.214 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:47.989 INFO:teuthology.orchestra.run.smithi036.stdout: "time": "2020-10-29T09:28:01.262778+0000", , 2020-10-29T09:30:47.989 INFO:teuthology.orchestra.run.smithi036.stdout: "duration": 1.01e-05 , 2020-10-29T09:30:47.989 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2020-10-29T22:44:42.215 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:48.003 INFO:teuthology.orchestra.run.smithi036.stdout: "time": "2020-10-29T09:28:05.298737+0000", , 2020-10-29T09:30:48.003 INFO:teuthology.orchestra.run.smithi036.stdout: "duration": 1.3937000000000001e-05 , 2020-10-29T09:30:48.003 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2020-10-29T22:44:42.216 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:48.018 INFO:teuthology.orchestra.run.smithi036.stdout: "duration": 0.021053715000000001, , 2020-10-29T09:30:48.018 INFO:teuthology.orchestra.run.smithi036.stdout: "type_data": { , 2020-10-29T09:30:48.019 INFO:teuthology.orchestra.run.smithi036.stdout: "flag_point": "started", 2020-10-29T22:44:42.216 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:48.032 INFO:teuthology.orchestra.run.smithi036.stdout: "event": "header_read", , 2020-10-29T09:30:48.033 INFO:teuthology.orchestra.run.smithi036.stdout: "time": "2020-10-29T09:28:34.477447+0000", , 2020-10-29T09:30:48.033 INFO:teuthology.orchestra.run.smithi036.stdout: "duration": 7.8779999999999998e-06 2020-10-29T22:44:42.217 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:48.046 INFO:teuthology.orchestra.run.smithi036.stdout: "event": "reached_pg", , 2020-10-29T09:30:48.047 INFO:teuthology.orchestra.run.smithi036.stdout: "time": "2020-10-29T09:28:34.483755+0000", , 2020-10-29T09:30:48.047 INFO:teuthology.orchestra.run.smithi036.stdout: "duration": 0.00013492699999999999 2020-10-29T22:44:42.217 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:48.061 INFO:teuthology.orchestra.run.smithi036.stdout: { , 2020-10-29T09:30:48.061 INFO:teuthology.orchestra.run.smithi036.stdout: "event": "initiated", , 2020-10-29T09:30:48.061 INFO:teuthology.orchestra.run.smithi036.stdout: "time": "2020-10-29T09:28:34.477736+0000", 2020-10-29T22:44:42.218 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:48.074 INFO:teuthology.orchestra.run.smithi036.stdout: { , 2020-10-29T09:30:48.075 INFO:teuthology.orchestra.run.smithi036.stdout: "event": "reached_pg", , 2020-10-29T09:30:48.075 INFO:teuthology.orchestra.run.smithi036.stdout: "time": "2020-10-29T09:28:44.563431+0000", 2020-10-29T22:44:42.218 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:48.088 INFO:teuthology.orchestra.run.smithi036.stdout: "initiated_at": "2020-10-29T09:28:44.563673+0000", , 2020-10-29T09:30:48.089 INFO:teuthology.orchestra.run.smithi036.stdout: "age": 123.35763878100001, , 2020-10-29T09:30:48.089 INFO:teuthology.orchestra.run.smithi036.stdout: "duration": 0.024663582, 2020-10-29T22:44:42.220 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:48.103 INFO:teuthology.orchestra.run.smithi036.stdout: }, , 2020-10-29T09:30:48.103 INFO:teuthology.orchestra.run.smithi036.stdout: { , 2020-10-29T09:30:48.103 INFO:teuthology.orchestra.run.smithi036.stdout: "event": "dispatched", 2020-10-29T22:44:42.221 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:48.116 INFO:teuthology.orchestra.run.smithi036.stdout: }, , 2020-10-29T09:30:48.117 INFO:teuthology.orchestra.run.smithi036.stdout: { , 2020-10-29T09:30:48.117 INFO:teuthology.orchestra.run.smithi036.stdout: "event": "sub_op_committed", 2020-10-29T22:44:42.221 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:48.130 INFO:teuthology.orchestra.run.smithi036.stdout: "flag_point": "started", , 2020-10-29T09:30:48.130 INFO:teuthology.orchestra.run.smithi036.stdout: "client_info": { , 2020-10-29T09:30:48.131 INFO:teuthology.orchestra.run.smithi036.stdout: "client": "client.4409", 2020-10-29T22:44:42.222 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:48.144 INFO:teuthology.orchestra.run.smithi036.stdout: "duration": 8.2570000000000007e-06 , 2020-10-29T09:30:48.144 INFO:teuthology.orchestra.run.smithi036.stdout: }, , 2020-10-29T09:30:48.144 INFO:teuthology.orchestra.run.smithi036.stdout: { 2020-10-29T22:44:42.222 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:48.545 INFO:teuthology.orchestra.run.smithi036.stdout: { , 2020-10-29T09:30:48.545 INFO:teuthology.orchestra.run.smithi036.stdout: "event": "initiated", , 2020-10-29T09:30:48.546 INFO:teuthology.orchestra.run.smithi036.stdout: "time": "2020-10-29T09:27:25.338561+0000", 2020-10-29T22:44:42.223 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:48.560 INFO:teuthology.orchestra.run.smithi036.stdout: { , 2020-10-29T09:30:48.560 INFO:teuthology.orchestra.run.smithi036.stdout: "event": "all_read", , 2020-10-29T09:30:48.560 INFO:teuthology.orchestra.run.smithi036.stdout: "time": "2020-10-29T09:27:25.339104+0000", 2020-10-29T22:44:42.223 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:48.575 INFO:teuthology.orchestra.run.smithi036.stdout: { , 2020-10-29T09:30:48.575 INFO:teuthology.orchestra.run.smithi036.stdout: "event": "queued_for_pg", , 2020-10-29T09:30:48.575 INFO:teuthology.orchestra.run.smithi036.stdout: "time": "2020-10-29T09:27:55.297286+0000", 2020-10-29T22:44:42.224 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:48.590 INFO:teuthology.orchestra.run.smithi036.stdout: { , 2020-10-29T09:30:48.590 INFO:teuthology.orchestra.run.smithi036.stdout: "event": "reached_pg", , 2020-10-29T09:30:48.591 INFO:teuthology.orchestra.run.smithi036.stdout: "time": "2020-10-29T09:28:20.368479+0000", 2020-10-29T22:44:42.224 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:48.605 INFO:teuthology.orchestra.run.smithi036.stdout: { , 2020-10-29T09:30:48.605 INFO:teuthology.orchestra.run.smithi036.stdout: "event": "started", , 2020-10-29T09:30:48.606 INFO:teuthology.orchestra.run.smithi036.stdout: "time": "2020-10-29T09:28:20.379889+0000", 2020-10-29T22:44:42.225 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:48.619 INFO:teuthology.orchestra.run.smithi036.stdout: "initiated_at": "2020-10-29T09:28:20.360312+0000", , 2020-10-29T09:30:48.619 INFO:teuthology.orchestra.run.smithi036.stdout: "age": 148.18052995900001, , 2020-10-29T09:30:48.620 INFO:teuthology.orchestra.run.smithi036.stdout: "duration": 0.028074323000000002, 2020-10-29T22:44:42.225 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:48.633 INFO:teuthology.orchestra.run.smithi036.stdout: "events": [ , 2020-10-29T09:30:48.634 INFO:teuthology.orchestra.run.smithi036.stdout: { , 2020-10-29T09:30:48.634 INFO:teuthology.orchestra.run.smithi036.stdout: "event": "initiated", 2020-10-29T22:44:42.226 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:48.647 INFO:teuthology.orchestra.run.smithi036.stdout: }, , 2020-10-29T09:30:48.648 INFO:teuthology.orchestra.run.smithi036.stdout: { , 2020-10-29T09:30:48.648 INFO:teuthology.orchestra.run.smithi036.stdout: "event": "reached_pg", 2020-10-29T22:44:42.226 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:48.661 INFO:teuthology.orchestra.run.smithi036.stdout: "description": "osd_op(client.4409.0:1099 3.9s0 3:98700ad4:::smithi03617554-47:39 [read 0~3489792,checksum] snapc 0=[] ondisk+read+known_if_redirected e100)", , 2020-10-29T09:30:48.662 INFO:teuthology.orchestra.run.smithi036.stdout: "initiated_at": "2020-10-29T09:28:35.484484+0000", , 2020-10-29T09:30:48.662 INFO:teuthology.orchestra.run.smithi036.stdout: "age": 133.056357486, 2020-10-29T22:44:42.227 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:48.675 INFO:teuthology.orchestra.run.smithi036.stdout: "duration": 3.1612999999999997e-05 , 2020-10-29T09:30:48.676 INFO:teuthology.orchestra.run.smithi036.stdout: }, , 2020-10-29T09:30:48.676 INFO:teuthology.orchestra.run.smithi036.stdout: { 2020-10-29T22:44:42.227 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:48.689 INFO:teuthology.orchestra.run.smithi036.stdout: "duration": 4.6653999999999998e-05 , 2020-10-29T09:30:48.689 INFO:teuthology.orchestra.run.smithi036.stdout: }, , 2020-10-29T09:30:48.689 INFO:teuthology.orchestra.run.smithi036.stdout: { 2020-10-29T22:44:42.228 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:48.703 INFO:teuthology.orchestra.run.smithi036.stdout: "duration": 0.014355994 , 2020-10-29T09:30:48.703 INFO:teuthology.orchestra.run.smithi036.stdout: }, , 2020-10-29T09:30:48.704 INFO:teuthology.orchestra.run.smithi036.stdout: { 2020-10-29T22:44:42.228 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:48.717 INFO:teuthology.orchestra.run.smithi036.stdout: "time": "2020-10-29T09:28:54.325460+0000", , 2020-10-29T09:30:48.717 INFO:teuthology.orchestra.run.smithi036.stdout: "duration": 5.8060000000000003e-05 , 2020-10-29T09:30:48.717 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2020-10-29T22:44:42.234 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:48.731 INFO:teuthology.orchestra.run.smithi036.stdout: }, , 2020-10-29T09:30:48.731 INFO:teuthology.orchestra.run.smithi036.stdout: { , 2020-10-29T09:30:48.731 INFO:teuthology.orchestra.run.smithi036.stdout: "description": "osd_op(client.4409.0:1292 3.ds0 3:b39a030d:::smithi03617554-24:53 [read 0~2408448,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e119)", 2020-10-29T22:44:42.235 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:48.745 INFO:teuthology.orchestra.run.smithi036.stdout: "event": "queued_for_pg", , 2020-10-29T09:30:48.745 INFO:teuthology.orchestra.run.smithi036.stdout: "time": "2020-10-29T09:28:54.330483+0000", , 2020-10-29T09:30:48.745 INFO:teuthology.orchestra.run.smithi036.stdout: "duration": 0.012323448000000001 2020-10-29T22:44:42.235 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:49.080 INFO:teuthology.orchestra.run.smithi013.stdout:} , 2020-10-29T09:30:49.089 INFO:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd reweight 0 1 , 2020-10-29T09:30:49.145 INFO:teuthology.orchestra.run.smithi013.stdout:{ 2020-10-29T22:44:42.236 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:49.161 INFO:teuthology.orchestra.run.smithi013.stdout: "client_addr": "172.21.15.36:0/4277591925", , 2020-10-29T09:30:49.161 INFO:teuthology.orchestra.run.smithi013.stdout: "tid": 425 , 2020-10-29T09:30:49.161 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2020-10-29T22:44:42.236 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:49.180 INFO:teuthology.orchestra.run.smithi013.stdout: "event": "header_read", , 2020-10-29T09:30:49.180 INFO:teuthology.orchestra.run.smithi013.stdout: "time": "2020-10-29T09:27:36.212184+0000", , 2020-10-29T09:30:49.180 INFO:teuthology.orchestra.run.smithi013.stdout: "duration": 1.0913e-05 2020-10-29T22:44:42.237 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:49.194 INFO:teuthology.orchestra.run.smithi013.stdout: "event": "done", , 2020-10-29T09:30:49.194 INFO:teuthology.orchestra.run.smithi013.stdout: "time": "2020-10-29T09:27:56.218469+0000", , 2020-10-29T09:30:49.195 INFO:teuthology.orchestra.run.smithi013.stdout: "duration": 0.90909471799999997 2020-10-29T22:44:42.237 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:49.209 INFO:teuthology.orchestra.run.smithi013.stdout: { , 2020-10-29T09:30:49.209 INFO:teuthology.orchestra.run.smithi013.stdout: "event": "header_read", , 2020-10-29T09:30:49.209 INFO:teuthology.orchestra.run.smithi013.stdout: "time": "2020-10-29T09:28:25.380346+0000", 2020-10-29T22:44:42.237 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:49.227 INFO:teuthology.orchestra.run.smithi013.stdout: { , 2020-10-29T09:30:49.228 INFO:teuthology.orchestra.run.smithi013.stdout: "event": "queued_for_pg", , 2020-10-29T09:30:49.228 INFO:teuthology.orchestra.run.smithi013.stdout: "time": "2020-10-29T09:28:31.445287+0000", 2020-10-29T22:44:42.240 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:49.243 INFO:teuthology.orchestra.run.smithi013.stdout: { , 2020-10-29T09:30:49.243 INFO:teuthology.orchestra.run.smithi013.stdout: "event": "reached_pg", , 2020-10-29T09:30:49.243 INFO:teuthology.orchestra.run.smithi013.stdout: "time": "2020-10-29T09:28:31.449405+0000", 2020-10-29T22:44:42.240 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:49.257 INFO:teuthology.orchestra.run.smithi013.stdout: "initiated_at": "2020-10-29T09:28:36.499325+0000", , 2020-10-29T09:30:49.257 INFO:teuthology.orchestra.run.smithi013.stdout: "age": 132.642489122, , 2020-10-29T09:30:49.257 INFO:teuthology.orchestra.run.smithi013.stdout: "duration": 0.031073955, 2020-10-29T22:44:42.241 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:49.270 INFO:teuthology.orchestra.run.smithi013.stdout: }, , 2020-10-29T09:30:49.270 INFO:teuthology.orchestra.run.smithi013.stdout: { , 2020-10-29T09:30:49.270 INFO:teuthology.orchestra.run.smithi013.stdout: "event": "reached_pg", 2020-10-29T22:44:42.241 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:49.284 INFO:teuthology.orchestra.run.smithi013.stdout: "description": "osd_op(client.4409.0:1275 3.bs0 3:dd5ae458:::smithi03617554-34:head [append 0~405504] snapc 5c=[5c,5b,5a,53,50,4d,4b,33,30,27] ondisk+write+known_if_redirected e118)", , 2020-10-29T09:30:49.284 INFO:teuthology.orchestra.run.smithi013.stdout: "initiated_at": "2020-10-29T09:28:53.497142+0000", , 2020-10-29T09:30:49.284 INFO:teuthology.orchestra.run.smithi013.stdout: "age": 115.64467245500001, 2020-10-29T22:44:42.242 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:49.298 INFO:teuthology.orchestra.run.smithi013.stdout: "duration": 8.0887e-05 , 2020-10-29T09:30:49.298 INFO:teuthology.orchestra.run.smithi013.stdout: }, , 2020-10-29T09:30:49.298 INFO:teuthology.orchestra.run.smithi013.stdout: { 2020-10-29T22:44:42.242 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:49.311 INFO:teuthology.orchestra.run.smithi013.stdout: "duration": 0.0067902359999999998 , 2020-10-29T09:30:49.311 INFO:teuthology.orchestra.run.smithi013.stdout: }, , 2020-10-29T09:30:49.312 INFO:teuthology.orchestra.run.smithi013.stdout: { 2020-10-29T22:44:42.243 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:49.325 INFO:teuthology.orchestra.run.smithi013.stdout: "duration": 1.0214697530000001 , 2020-10-29T09:30:49.325 INFO:teuthology.orchestra.run.smithi013.stdout: }, , 2020-10-29T09:30:49.325 INFO:teuthology.orchestra.run.smithi013.stdout: { 2020-10-29T22:44:42.243 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:49.338 INFO:teuthology.orchestra.run.smithi013.stdout: "duration": 1.819265057 , 2020-10-29T09:30:49.339 INFO:teuthology.orchestra.run.smithi013.stdout: } , 2020-10-29T09:30:49.339 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2020-10-29T22:44:42.244 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:49.352 INFO:teuthology.orchestra.run.smithi013.stdout: "type_data": { , 2020-10-29T09:30:49.352 INFO:teuthology.orchestra.run.smithi013.stdout: "flag_point": "started", , 2020-10-29T09:30:49.352 INFO:teuthology.orchestra.run.smithi013.stdout: "client_info": { 2020-10-29T22:44:42.244 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:49.367 INFO:teuthology.orchestra.run.smithi013.stdout: "time": "2020-10-29T09:30:46.360342+0000", , 2020-10-29T09:30:49.367 INFO:teuthology.orchestra.run.smithi013.stdout: "duration": 8.6039999999999993e-06 , 2020-10-29T09:30:49.367 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2020-10-29T22:44:42.245 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:49.752 INFO:teuthology.orchestra.run.smithi013.stdout: "duration": 0.002549394 , 2020-10-29T09:30:49.752 INFO:teuthology.orchestra.run.smithi013.stdout: }, , 2020-10-29T09:30:49.752 INFO:teuthology.orchestra.run.smithi013.stdout: { 2020-10-29T22:44:42.245 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:49.765 INFO:teuthology.orchestra.run.smithi013.stdout: "time": "2020-10-29T09:30:36.442516+0000", , 2020-10-29T09:30:49.766 INFO:teuthology.orchestra.run.smithi013.stdout: "duration": 0.0024430720000000001 , 2020-10-29T09:30:49.766 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2020-10-29T22:44:42.246 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:49.779 INFO:teuthology.orchestra.run.smithi013.stdout: "event": "reached_pg", , 2020-10-29T09:30:49.779 INFO:teuthology.orchestra.run.smithi013.stdout: "time": "2020-10-29T09:30:36.516723+0000", , 2020-10-29T09:30:49.780 INFO:teuthology.orchestra.run.smithi013.stdout: "duration": 1.0419e-05 2020-10-29T22:44:42.246 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:49.796 INFO:teuthology.orchestra.run.smithi013.stdout: { , 2020-10-29T09:30:49.796 INFO:teuthology.orchestra.run.smithi013.stdout: "event": "dispatched", , 2020-10-29T09:30:49.796 INFO:teuthology.orchestra.run.smithi013.stdout: "time": "2020-10-29T09:30:36.543468+0000", 2020-10-29T22:44:42.247 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:49.809 INFO:teuthology.orchestra.run.smithi013.stdout: }, , 2020-10-29T09:30:49.809 INFO:teuthology.orchestra.run.smithi013.stdout: { , 2020-10-29T09:30:49.810 INFO:teuthology.orchestra.run.smithi013.stdout: "event": "header_read", 2020-10-29T22:44:42.247 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:49.823 INFO:teuthology.orchestra.run.smithi013.stdout: "duration": 0.000197126 , 2020-10-29T09:30:49.823 INFO:teuthology.orchestra.run.smithi013.stdout: }, , 2020-10-29T09:30:49.823 INFO:teuthology.orchestra.run.smithi013.stdout: { 2020-10-29T22:44:42.248 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:49.837 INFO:teuthology.orchestra.run.smithi013.stdout: "type_data": { , 2020-10-29T09:30:49.837 INFO:teuthology.orchestra.run.smithi013.stdout: "flag_point": "reached pg", , 2020-10-29T09:30:49.837 INFO:teuthology.orchestra.run.smithi013.stdout: "events": [ 2020-10-29T22:44:42.248 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:49.851 INFO:teuthology.orchestra.run.smithi013.stdout: "time": "2020-10-29T09:30:47.428150+0000", , 2020-10-29T09:30:49.851 INFO:teuthology.orchestra.run.smithi013.stdout: "duration": 0.002498779 , 2020-10-29T09:30:49.851 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2020-10-29T22:44:42.249 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:49.864 INFO:teuthology.orchestra.run.smithi013.stdout: "event": "queued_for_pg", , 2020-10-29T09:30:49.864 INFO:teuthology.orchestra.run.smithi013.stdout: "time": "2020-10-29T09:30:47.449280+0000", , 2020-10-29T09:30:49.864 INFO:teuthology.orchestra.run.smithi013.stdout: "duration": 0.00059027500000000002 2020-10-29T22:44:42.249 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:49.878 INFO:teuthology.orchestra.run.smithi013.stdout: { , 2020-10-29T09:30:49.878 INFO:teuthology.orchestra.run.smithi013.stdout: "event": "throttled", , 2020-10-29T09:30:49.878 INFO:teuthology.orchestra.run.smithi013.stdout: "time": "2020-10-29T09:30:47.451361+0000", 2020-10-29T22:44:42.250 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:49.891 INFO:teuthology.orchestra.run.smithi013.stdout: { , 2020-10-29T09:30:49.892 INFO:teuthology.orchestra.run.smithi013.stdout: "event": "done", , 2020-10-29T09:30:49.892 INFO:teuthology.orchestra.run.smithi013.stdout: "time": "2020-10-29T09:30:47.493553+0000", 2020-10-29T22:44:42.250 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:50.300 INFO:teuthology.orchestra.run.smithi036.stdout: "event": "throttled", , 2020-10-29T09:30:50.301 INFO:teuthology.orchestra.run.smithi036.stdout: "time": "2020-10-29T09:30:36.293504+0000", , 2020-10-29T09:30:50.301 INFO:teuthology.orchestra.run.smithi036.stdout: "duration": 5.4017000000000001e-05 2020-10-29T22:44:42.251 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:50.330 INFO:teuthology.orchestra.run.smithi036.stdout: "event": "done", , 2020-10-29T09:30:50.330 INFO:teuthology.orchestra.run.smithi036.stdout: "time": "2020-10-29T09:30:36.301828+0000", , 2020-10-29T09:30:50.330 INFO:teuthology.orchestra.run.smithi036.stdout: "duration": 0.0031661229999999999 2020-10-29T22:44:42.251 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:50.347 INFO:teuthology.orchestra.run.smithi036.stdout: { , 2020-10-29T09:30:50.347 INFO:teuthology.orchestra.run.smithi036.stdout: "event": "queued_for_pg", , 2020-10-29T09:30:50.347 INFO:teuthology.orchestra.run.smithi036.stdout: "time": "2020-10-29T09:30:36.520056+0000", 2020-10-29T22:44:42.252 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:50.362 INFO:teuthology.orchestra.run.smithi036.stdout: }, , 2020-10-29T09:30:50.362 INFO:teuthology.orchestra.run.smithi036.stdout: { , 2020-10-29T09:30:50.363 INFO:teuthology.orchestra.run.smithi036.stdout: "event": "throttled", 2020-10-29T22:44:42.252 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:50.383 INFO:teuthology.orchestra.run.smithi036.stdout: }, , 2020-10-29T09:30:50.383 INFO:teuthology.orchestra.run.smithi036.stdout: { , 2020-10-29T09:30:50.383 INFO:teuthology.orchestra.run.smithi036.stdout: "event": "done", 2020-10-29T22:44:42.253 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:50.398 INFO:teuthology.orchestra.run.smithi036.stdout: "duration": 6.7259999999999996e-06 , 2020-10-29T09:30:50.398 INFO:teuthology.orchestra.run.smithi036.stdout: }, , 2020-10-29T09:30:50.398 INFO:teuthology.orchestra.run.smithi036.stdout: { 2020-10-29T22:44:42.253 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:50.412 INFO:teuthology.orchestra.run.smithi036.stdout: "time": "2020-10-29T09:30:36.547387+0000", , 2020-10-29T09:30:50.412 INFO:teuthology.orchestra.run.smithi036.stdout: "duration": 0 , 2020-10-29T09:30:50.412 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2020-10-29T22:44:42.254 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:50.430 INFO:teuthology.orchestra.run.smithi036.stdout: "time": "2020-10-29T09:30:36.555790+0000", , 2020-10-29T09:30:50.430 INFO:teuthology.orchestra.run.smithi036.stdout: "duration": 5.5241000000000002e-05 , 2020-10-29T09:30:50.430 INFO:teuthology.orchestra.run.smithi036.stdout: }, 2020-10-29T22:44:42.254 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:50.446 INFO:teuthology.orchestra.run.smithi036.stdout: "event": "dispatched", , 2020-10-29T09:30:50.446 INFO:teuthology.orchestra.run.smithi036.stdout: "time": "2020-10-29T09:30:36.554711+0000", , 2020-10-29T09:30:50.446 INFO:teuthology.orchestra.run.smithi036.stdout: "duration": 4.8239999999999999e-06 2020-10-29T22:44:42.254 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:50.462 INFO:teuthology.orchestra.run.smithi036.stdout: { , 2020-10-29T09:30:50.462 INFO:teuthology.orchestra.run.smithi036.stdout: "event": "initiated", , 2020-10-29T09:30:50.462 INFO:teuthology.orchestra.run.smithi036.stdout: "time": "2020-10-29T09:30:36.569522+0000", 2020-10-29T22:44:42.255 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:50.477 INFO:teuthology.orchestra.run.smithi036.stdout: { , 2020-10-29T09:30:50.477 INFO:teuthology.orchestra.run.smithi036.stdout: "event": "sub_op_committed", , 2020-10-29T09:30:50.477 INFO:teuthology.orchestra.run.smithi036.stdout: "time": "2020-10-29T09:30:36.577119+0000", 2020-10-29T22:44:42.256 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:50.492 INFO:teuthology.orchestra.run.smithi036.stdout: }, , 2020-10-29T09:30:50.492 INFO:teuthology.orchestra.run.smithi036.stdout: { , 2020-10-29T09:30:50.492 INFO:teuthology.orchestra.run.smithi036.stdout: "event": "dispatched", 2020-10-29T22:44:42.257 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:50.510 INFO:teuthology.orchestra.run.smithi013:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight , 2020-10-29T09:30:50.637 INFO:teuthology.orchestra.run.smithi013.stdout:{ , 2020-10-29T09:30:50.638 INFO:teuthology.orchestra.run.smithi013.stdout: "ops": [], 2020-10-29T22:44:42.257 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:50.899 INFO:teuthology.orchestra.run.smithi013.stdout: "event": "dispatched", , 2020-10-29T09:30:50.899 INFO:teuthology.orchestra.run.smithi013.stdout: "time": "2020-10-29T09:30:42.584235+0000", , 2020-10-29T09:30:50.899 INFO:teuthology.orchestra.run.smithi013.stdout: "duration": 1.7560000000000001e-05 2020-10-29T22:44:42.258 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:50.917 INFO:teuthology.orchestra.run.smithi013.stdout: { , 2020-10-29T09:30:50.917 INFO:teuthology.orchestra.run.smithi013.stdout: "event": "dispatched", , 2020-10-29T09:30:50.917 INFO:teuthology.orchestra.run.smithi013.stdout: "time": "2020-10-29T09:30:42.656074+0000", 2020-10-29T22:44:42.258 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:50.931 INFO:teuthology.orchestra.run.smithi013.stdout: }, , 2020-10-29T09:30:50.931 INFO:teuthology.orchestra.run.smithi013.stdout: { , 2020-10-29T09:30:50.931 INFO:teuthology.orchestra.run.smithi013.stdout: "event": "dispatched", 2020-10-29T22:44:42.259 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:50.945 INFO:teuthology.orchestra.run.smithi013.stdout: "duration": 1.0403e-05 , 2020-10-29T09:30:50.946 INFO:teuthology.orchestra.run.smithi013.stdout: }, , 2020-10-29T09:30:50.946 INFO:teuthology.orchestra.run.smithi013.stdout: { 2020-10-29T22:44:42.259 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:50.959 INFO:teuthology.orchestra.run.smithi013.stdout: "time": "2020-10-29T09:30:42.690030+0000", , 2020-10-29T09:30:50.959 INFO:teuthology.orchestra.run.smithi013.stdout: "duration": 1.0482999999999999e-05 , 2020-10-29T09:30:50.959 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2020-10-29T22:44:42.260 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:50.973 INFO:teuthology.orchestra.run.smithi013.stdout: "event": "all_read", , 2020-10-29T09:30:50.974 INFO:teuthology.orchestra.run.smithi013.stdout: "time": "2020-10-29T09:30:43.489986+0000", , 2020-10-29T09:30:50.974 INFO:teuthology.orchestra.run.smithi013.stdout: "duration": 6.0469e-05 2020-10-29T22:44:42.260 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:50.987 INFO:teuthology.orchestra.run.smithi013.stdout: { , 2020-10-29T09:30:50.987 INFO:teuthology.orchestra.run.smithi013.stdout: "event": "throttled", , 2020-10-29T09:30:50.987 INFO:teuthology.orchestra.run.smithi013.stdout: "time": "2020-10-29T09:30:47.423132+0000", 2020-10-29T22:44:42.261 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:51.001 INFO:teuthology.orchestra.run.smithi013.stdout: { , 2020-10-29T09:30:51.001 INFO:teuthology.orchestra.run.smithi013.stdout: "event": "done", , 2020-10-29T09:30:51.001 INFO:teuthology.orchestra.run.smithi013.stdout: "time": "2020-10-29T09:30:47.432116+0000", 2020-10-29T22:44:42.261 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:51.015 INFO:teuthology.orchestra.run.smithi013.stdout: }, , 2020-10-29T09:30:51.015 INFO:teuthology.orchestra.run.smithi013.stdout: { , 2020-10-29T09:30:51.015 INFO:teuthology.orchestra.run.smithi013.stdout: "event": "queued_for_pg", 2020-10-29T22:44:42.261 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:51.029 INFO:teuthology.orchestra.run.smithi013.stdout: "duration": 0 , 2020-10-29T09:30:51.029 INFO:teuthology.orchestra.run.smithi013.stdout: }, , 2020-10-29T09:30:51.029 INFO:teuthology.orchestra.run.smithi013.stdout: { 2020-10-29T22:44:42.262 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:51.042 INFO:teuthology.orchestra.run.smithi013.stdout: "duration": 3.4258999999999998e-05 , 2020-10-29T09:30:51.043 INFO:teuthology.orchestra.run.smithi013.stdout: }, , 2020-10-29T09:30:51.043 INFO:teuthology.orchestra.run.smithi013.stdout: { 2020-10-29T22:44:42.262 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:51.474 INFO:teuthology.orchestra.run.smithi013.stdout: "time": "2020-10-29T09:30:47.441484+0000", , 2020-10-29T09:30:51.474 INFO:teuthology.orchestra.run.smithi013.stdout: "duration": 0.0012588429999999999 , 2020-10-29T09:30:51.474 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2020-10-29T22:44:42.263 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:51.493 INFO:teuthology.orchestra.run.smithi013.stdout: "time": "2020-10-29T09:30:47.447400+0000", , 2020-10-29T09:30:51.493 INFO:teuthology.orchestra.run.smithi013.stdout: "duration": 0.0063930109999999997 , 2020-10-29T09:30:51.493 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2020-10-29T22:44:42.263 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:51.513 INFO:teuthology.orchestra.run.smithi013.stdout: "time": "2020-10-29T09:30:47.453891+0000", , 2020-10-29T09:30:51.513 INFO:teuthology.orchestra.run.smithi013.stdout: "duration": 0.001582209 , 2020-10-29T09:30:51.513 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2020-10-29T22:44:42.264 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:51.526 INFO:teuthology.orchestra.run.smithi013.stdout: "time": "2020-10-29T09:30:47.462754+0000", , 2020-10-29T09:30:51.527 INFO:teuthology.orchestra.run.smithi013.stdout: "duration": 0.87695853199999996 , 2020-10-29T09:30:51.527 INFO:teuthology.orchestra.run.smithi013.stdout: } 2020-10-29T22:44:42.264 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:51.540 INFO:teuthology.orchestra.run.smithi013.stdout: }, , 2020-10-29T09:30:51.540 INFO:teuthology.orchestra.run.smithi013.stdout: { , 2020-10-29T09:30:51.540 INFO:teuthology.orchestra.run.smithi013.stdout: "description": "osd_op(client.4409.0:1307 3.4s0 3:27cd746a:::smithi03617554-50:head [append 0~405504] snapc 60=[60,5c,5a,53,50,4d,4b,33,30,27] RETRY=5 ondisk+retry+write+known_if_redirected e175)", 2020-10-29T22:44:42.265 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:51.554 INFO:teuthology.orchestra.run.smithi013.stdout: "duration": 0.84975306699999997, , 2020-10-29T09:30:51.554 INFO:teuthology.orchestra.run.smithi013.stdout: "type_data": { , 2020-10-29T09:30:51.554 INFO:teuthology.orchestra.run.smithi013.stdout: "flag_point": "commit sent; apply or cleanup", 2020-10-29T22:44:42.265 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:51.568 INFO:teuthology.orchestra.run.smithi013.stdout: "client_addr": "172.21.15.36:0/4277591925", , 2020-10-29T09:30:51.568 INFO:teuthology.orchestra.run.smithi013.stdout: "tid": 1309 , 2020-10-29T09:30:51.568 INFO:teuthology.orchestra.run.smithi013.stdout: }, 2020-10-29T22:44:42.265 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:51.581 INFO:teuthology.orchestra.run.smithi013.stdout: "event": "header_read", , 2020-10-29T09:30:51.581 INFO:teuthology.orchestra.run.smithi013.stdout: "time": "2020-10-29T09:30:47.375453+0000", , 2020-10-29T09:30:51.582 INFO:teuthology.orchestra.run.smithi013.stdout: "duration": 2.7586999999999999e-05 2020-10-29T22:44:42.266 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:51.595 INFO:teuthology.orchestra.run.smithi013.stdout: { , 2020-10-29T09:30:51.595 INFO:teuthology.orchestra.run.smithi013.stdout: "event": "initiated", , 2020-10-29T09:30:51.595 INFO:teuthology.orchestra.run.smithi013.stdout: "time": "2020-10-29T09:30:47.420804+0000", 2020-10-29T22:44:42.266 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:51.611 INFO:teuthology.orchestra.run.smithi013.stdout: "events": [ , 2020-10-29T09:30:51.611 INFO:teuthology.orchestra.run.smithi013.stdout: { , 2020-10-29T09:30:51.611 INFO:teuthology.orchestra.run.smithi013.stdout: "event": "initiated", 2020-10-29T22:44:42.267 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:51.632 INFO:teuthology.orchestra.run.smithi013.stdout: "flag_point": "reached pg", , 2020-10-29T09:30:51.632 INFO:teuthology.orchestra.run.smithi013.stdout: "events": [ , 2020-10-29T09:30:51.632 INFO:teuthology.orchestra.run.smithi013.stdout: { 2020-10-29T22:44:42.267 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:51.647 INFO:teuthology.orchestra.run.smithi013.stdout: "duration": 0.0043636639999999997 , 2020-10-29T09:30:51.647 INFO:teuthology.orchestra.run.smithi013.stdout: } , 2020-10-29T09:30:51.648 INFO:teuthology.orchestra.run.smithi013.stdout: ] 2020-10-29T22:44:42.268 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:51.663 INFO:teuthology.orchestra.run.smithi013.stdout: "time": "2020-10-29T09:30:47.452751+0000", , 2020-10-29T09:30:51.663 INFO:teuthology.orchestra.run.smithi013.stdout: "duration": 0.0079634600000000003 , 2020-10-29T09:30:51.663 INFO:teuthology.orchestra.run.smithi013.stdout: } 2020-10-29T22:44:42.271 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:30:51.680 INFO:teuthology.orchestra.run.smithi013.stdout: "event": "done", , 2020-10-29T09:30:51.681 INFO:teuthology.orchestra.run.smithi013.stdout: "time": "2020-10-29T09:30:47.464082+0000", , 2020-10-29T09:30:51.681 INFO:teuthology.orchestra.run.smithi013.stdout: "duration": 0.0053570099999999997 2020-10-29T22:44:42.273 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:31:25.794 INFO:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json , 2020-10-29T09:31:26.097 INFO:teuthology.orchestra.run.smithi036.stdout: , 2020-10-29T09:31:26.097 INFO:teuthology.orchestra.run.smithi036.stdout:{"epoch":208,"fsid":"38ef033b-12ee-4aad-8387-8e1d1388adb2","created":"2020-10-29T09:26:51.249573+0000","modified":"2020-10-29T09:31:25.761499+0000","last_up_change":"2020-10-29T09:30:46.339435+0000","last_in_change":"2020-10-29T09:26:58.301305+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-10-29T09:26:55.318029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"primary_write_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"159","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"108","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-10-29T09:27:02.341857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"primary_write_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":18,"pg_placement_num":18,"pg_placement_num_target":18,"pg_num_target":18,"pg_num_pending":18,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"67","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2020-10-29T09:27:15.241912+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":4,"min_size":3,"primary_write_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":26,"pg_placement_num":26,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":26,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"191","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"180","auid":0,"snap_mode":"selfmanaged","snap_seq":99,"snap_epoch":122,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"jerasure31profile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":12288,"expected_num_objects":0,"fast_read":true,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"21110ae5-d05b-48cc-968e-76a63d387aba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6824","nonce":13961},{"type":"v1","addr":"172.21.15.36:6825","nonce":13961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6826","nonce":13961},{"type":"v1","addr":"172.21.15.36:6827","nonce":13961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6830","nonce":13961},{"type":"v1","addr":"172.21.15.36:6831","nonce":13961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6828","nonce":13961},{"type":"v1","addr":"172.21.15.36:6829","nonce":13961}]},"public_addr":"172.21.15.36:6825/13961","cluster_addr":"172.21.15.36:6827/13961","heartbeat_back_addr":"172.21.15.36:6831/13961","heartbeat_front_addr":"172.21.15.36:6829/13961","state":["exists","up"]},{"osd":1,"uuid":"0d798323-b11f-4529-8dfb-c8df17ed7321","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":144,"up_from":166,"up_thru":185,"down_at":145,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6800","nonce":29089},{"type":"v1","addr":"172.21.15.36:6801","nonce":29089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6802","nonce":29089},{"type":"v1","addr":"172.21.15.36:6803","nonce":29089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6806","nonce":29089},{"type":"v1","addr":"172.21.15.36:6807","nonce":29089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6804","nonce":29089},{"type":"v1","addr":"172.21.15.36:6805","nonce":29089}]},"public_addr":"172.21.15.36:6801/29089","cluster_addr":"172.21.15.36:6803/29089","heartbeat_back_addr":"172.21.15.36:6807/29089","heartbeat_front_addr":"172.21.15.36:6805/29089","state":["exists","up"]},{"osd":2,"uuid":"01935ec0-c80f-42c4-8194-af8bfc21cf52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6808","nonce":13943},{"type":"v1","addr":"172.21.15.36:6809","nonce":13943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6810","nonce":13943},{"type":"v1","addr":"172.21.15.36:6811","nonce":13943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6814","nonce":13943},{"type":"v1","addr":"172.21.15.36:6815","nonce":13943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6812","nonce":13943},{"type":"v1","addr":"172.21.15.36:6813","nonce":13943}]},"public_addr":"172.21.15.36:6809/13943","cluster_addr":"172.21.15.36:6811/13943","heartbeat_back_addr":"172.21.15.36:6815/13943","heartbeat_front_addr":"172.21.15.36:6813/13943","state":["exists","up"]},{"osd":3,"uuid":"1b1a9699-10a7-406b-ae7c-879bb4d7580a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":185,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6816","nonce":13944},{"type":"v1","addr":"172.21.15.36:6817","nonce":13944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6818","nonce":13944},{"type":"v1","addr":"172.21.15.36:6819","nonce":13944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6822","nonce":13944},{"type":"v1","addr":"172.21.15.36:6823","nonce":13944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6820","nonce":13944},{"type":"v1","addr":"172.21.15.36:6821","nonce":13944}]},"public_addr":"172.21.15.36:6817/13944","cluster_addr":"172.21.15.36:6819/13944","heartbeat_back_addr":"172.21.15.36:6823/13944","heartbeat_front_addr":"172.21.15.36:6821/13944","state":["exists","up"]},{"osd":4,"uuid":"c037dd95-a7d1-40a5-9462-744e6cab39af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":76,"up_from":163,"up_thru":185,"down_at":77,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6800","nonce":31590},{"type":"v1","addr":"172.21.15.13:6801","nonce":31590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6802","nonce":31590},{"type":"v1","addr":"172.21.15.13:6803","nonce":31590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6806","nonce":31590},{"type":"v1","addr":"172.21.15.13:6807","nonce":31590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6804","nonce":31590},{"type":"v1","addr":"172.21.15.13:6805","nonce":31590}]},"public_addr":"172.21.15.13:6801/31590","cluster_addr":"172.21.15.13:6803/31590","heartbeat_back_addr":"172.21.15.13:6807/31590","heartbeat_front_addr":"172.21.15.13:6805/31590","state":["exists","up"]},{"osd":5,"uuid":"ef943701-8f4d-422c-8f47-3979cd549284","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":136,"up_from":171,"up_thru":182,"down_at":137,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6808","nonce":32260},{"type":"v1","addr":"172.21.15.13:6809","nonce":32260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6810","nonce":32260},{"type":"v1","addr":"172.21.15.13:6811","nonce":32260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6814","nonce":32260},{"type":"v1","addr":"172.21.15.13:6815","nonce":32260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6812","nonce":32260},{"type":"v1","addr":"172.21.15.13:6813","nonce":32260}]},"public_addr":"172.21.15.13:6809/32260","cluster_addr":"172.21.15.13:6811/32260","heartbeat_back_addr":"172.21.15.13:6815/32260","heartbeat_front_addr":"172.21.15.13:6813/32260","state":["exists","up"]},{"osd":6,"uuid":"e51a88a5-4df1-410c-bb96-fdeeb5600302","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":117,"up_from":175,"up_thru":185,"down_at":119,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6824","nonce":32880},{"type":"v1","addr":"172.21.15.13:6825","nonce":32880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6826","nonce":32880},{"type":"v1","addr":"172.21.15.13:6827","nonce":32880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6830","nonce":32880},{"type":"v1","addr":"172.21.15.13:6831","nonce":32880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6828","nonce":32880},{"type":"v1","addr":"172.21.15.13:6829","nonce":32880}]},"public_addr":"172.21.15.13:6825/32880","cluster_addr":"172.21.15.13:6827/32880","heartbeat_back_addr":"172.21.15.13:6831/32880","heartbeat_front_addr":"172.21.15.13:6829/32880","state":["exists","up"]},{"osd":7,"uuid":"4f872205-5db9-42f0-be15-bf4a57855c6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6816","nonce":14026},{"type":"v1","addr":"172.21.15.13:6817","nonce":14026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6818","nonce":14026},{"type":"v1","addr":"172.21.15.13:6819","nonce":14026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6822","nonce":14026},{"type":"v1","addr":"172.21.15.13:6823","nonce":14026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.13:6820","nonce":14026},{"type":"v1","addr":"172.21.15.13:6821","nonce":14026}]},"public_addr":"172.21.15.13:6817/14026","cluster_addr":"172.21.15.13:6819/14026","heartbeat_back_addr":"172.21.15.13:6823/14026","heartbeat_front_addr":"172.21.15.13:6821/14026","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-29T09:30:59.541229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-29T09:30:36.071578+0000","dead_epoch":145},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-29T09:30:57.726096+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-29T09:30:58.116234+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-29T09:30:30.726086+0000","dead_epoch":77},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-29T09:30:41.523653+0000","dead_epoch":137},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-29T09:30:46.925793+0000","dead_epoch":119},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-29T09:30:58.762418+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.7","osds":[5,2]},{"pgid":"3.1","osds":[7,6,3,1]}],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":4,"to":6}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"jerasure31profile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"3","m":"1","name":"jerasure31profile","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-10-29T22:44:42.274 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:31:36.720 INFO:tasks.ceph:Scrubbing osd.4 , 2020-10-29T09:31:36.721 INFO:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.4 config set osd_debug_deep_scrub_sleep 0 , 2020-10-29T09:31:36.892 INFO:teuthology.orchestra.run.smithi036.stdout:{ 2020-10-29T22:44:42.277 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:31:39.078 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2020-10-29T09:30:49.843857+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=9, tm_min=30, tm_sec=49, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=9, tm_min=31, tm_sec=33, tm_wday=3, tm_yday=303, tm_isdst=0) , 2020-10-29T09:31:39.079 INFO:tasks.ceph:pgid 1.c last_scrub_stamp 2020-10-29T09:30:53.774004+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=9, tm_min=30, tm_sec=53, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=9, tm_min=31, tm_sec=33, tm_wday=3, tm_yday=303, tm_isdst=0) , 2020-10-29T09:31:39.079 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2020-10-29T09:30:56.804355+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=9, tm_min=30, tm_sec=56, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=9, tm_min=31, tm_sec=33, tm_wday=3, tm_yday=303, tm_isdst=0) 2020-10-29T22:44:42.277 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:32:41.117 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-29T09:32:41.143 INFO:tasks.ceph.mgr.x:Stopped , 2020-10-29T09:32:41.143 INFO:teuthology.misc:Shutting down mon daemons... 2020-10-29T22:44:42.278 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:32:58.488 INFO:teuthology.orchestra.run.smithi013:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits , 2020-10-29T09:32:58.495 INFO:teuthology.orchestra.run.smithi036:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits , 2020-10-29T09:32:58.562 DEBUG:teuthology.run_tasks:Unwinding manager clock 2020-10-29T22:44:43.099 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6268-g944bb62c (944bb62c829d723d3f38fa6a8966ac655e3866f2) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f9a1b787cfb] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x5633fc98b9b0] 2020-10-29T22:44:43.100 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:10:07.535 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", , 2020-10-29T09:10:07.535 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2020-10-29T09:05:06.081122+0000", , 2020-10-29T09:10:07.536 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 9.3000999999999999e-05 2020-10-29T22:44:43.101 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:10:07.567 INFO:teuthology.orchestra.run.smithi002.stdout: { , 2020-10-29T09:10:07.567 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", , 2020-10-29T09:10:07.567 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2020-10-29T09:05:51.784172+0000", 2020-10-29T22:44:43.102 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:10:07.601 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2020-10-29T09:10:07.602 INFO:teuthology.orchestra.run.smithi002.stdout: { , 2020-10-29T09:10:07.602 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "throttled", 2020-10-29T22:44:43.103 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:10:07.629 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2020-10-29T09:10:07.629 INFO:teuthology.orchestra.run.smithi002.stdout: { , 2020-10-29T09:10:07.629 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "waiting_for_map not empty", 2020-10-29T22:44:43.104 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:10:07.659 INFO:teuthology.orchestra.run.smithi002.stdout: "flag_point": "reached pg", , 2020-10-29T09:10:07.659 INFO:teuthology.orchestra.run.smithi002.stdout: "events": [ , 2020-10-29T09:10:07.659 INFO:teuthology.orchestra.run.smithi002.stdout: { 2020-10-29T22:44:43.104 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:10:07.702 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.136325422 , 2020-10-29T09:10:07.702 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2020-10-29T09:10:07.702 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2020-10-29T22:44:43.105 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:10:07.716 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2020-10-29T09:07:19.046079+0000", , 2020-10-29T09:10:07.716 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0019927370000000001 , 2020-10-29T09:10:07.717 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2020-10-29T22:44:43.106 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:10:07.738 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "queued_for_pg", , 2020-10-29T09:10:07.738 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2020-10-29T09:07:40.734672+0000", , 2020-10-29T09:10:07.738 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00016601699999999999 2020-10-29T22:44:43.107 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:10:07.754 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_started", , 2020-10-29T09:10:07.754 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2020-10-29T09:10:06.950751+0000", , 2020-10-29T09:10:07.754 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.0058618819999999997 2020-10-29T22:44:43.108 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:10:07.770 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "sub_op_committed", , 2020-10-29T09:10:07.770 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2020-10-29T09:10:06.961223+0000", , 2020-10-29T09:10:07.771 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.00021660900000000001 2020-10-29T22:44:43.108 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:10:07.786 INFO:teuthology.orchestra.run.smithi002.stdout: "event": "commit_sent", , 2020-10-29T09:10:07.786 INFO:teuthology.orchestra.run.smithi002.stdout: "time": "2020-10-29T09:10:06.975919+0000", , 2020-10-29T09:10:07.786 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.001535638 2020-10-29T22:44:43.109 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:10:07.807 INFO:teuthology.orchestra.run.smithi002.stdout: } , 2020-10-29T09:10:07.807 INFO:teuthology.orchestra.run.smithi002.stdout: }, , 2020-10-29T09:10:07.808 INFO:teuthology.orchestra.run.smithi002.stdout: { 2020-10-29T22:44:43.110 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:10:07.825 INFO:teuthology.orchestra.run.smithi002.stdout: "age": 0.35308520199999999, , 2020-10-29T09:10:07.825 INFO:teuthology.orchestra.run.smithi002.stdout: "duration": 0.232631755, , 2020-10-29T09:10:07.825 INFO:teuthology.orchestra.run.smithi002.stdout: "type_data": { 2020-10-29T22:44:43.111 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:10:08.197 INFO:teuthology.orchestra.run.smithi002.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. , 2020-10-29T09:10:08.208 INFO:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops , 2020-10-29T09:10:08.221 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4: {} 2020-10-29T22:44:43.113 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:10:20.249 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a: {} , 2020-10-29T09:10:20.266 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b: {} , 2020-10-29T09:10:20.286 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c: {} 2020-10-29T22:44:43.115 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:10:43.074 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2020-10-29T09:10:43.001995+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":389087232,"num_objects":265,"num_object_clones":213,"num_object_copies":1056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":265,"num_whiteouts":16,"num_read":1077,"num_read_kb":1039184,"num_write":2228,"num_write_kb":412730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":549,"num_bytes_recovered":766537728,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5142,"ondisk_log_size":5142,"up":122,"acting":122,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":122,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":1151148,"kb_used_data":576876,"kb_used_omap":218,"kb_used_meta":573989,"kb_avail":753823572,"statfs":{"total":773094113280,"available":771915337728,"internally_reserved":0,"allocated":590721024,"data_stored":925104048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":223260,"internal_metadata":587765732},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002780"},"pg_stats":[{"pgid":"2.1b","version":"0'0","reported_seq":"184","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:36.146327+0000","last_change":"2020-10-29T09:10:12.466861+0000","last_active":"2020-10-29T09:10:36.146327+0000","last_peered":"2020-10-29T09:10:36.146327+0000","last_clean":"2020-10-29T09:10:36.146327+0000","last_became_active":"2020-10-29T09:10:06.931997+0000","last_became_peered":"2020-10-29T09:10:06.931997+0000","last_unstale":"2020-10-29T09:10:36.146327+0000","last_undegraded":"2020-10-29T09:10:36.146327+0000","last_fullsized":"2020-10-29T09:10:36.146327+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-10-29T09:10:12.466651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:07:17.791921+0000","last_clean_scrub_stamp":"2020-10-29T09:10:12.466651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":"43","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:36.138138+0000","last_change":"2020-10-29T09:10:32.198641+0000","last_active":"2020-10-29T09:10:36.138138+0000","last_peered":"2020-10-29T09:10:36.138138+0000","last_clean":"2020-10-29T09:10:36.138138+0000","last_became_active":"2020-10-29T09:10:18.047556+0000","last_became_peered":"2020-10-29T09:10:18.047556+0000","last_unstale":"2020-10-29T09:10:36.138138+0000","last_undegraded":"2020-10-29T09:10:36.138138+0000","last_fullsized":"2020-10-29T09:10:36.138138+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-10-29T09:10:18.128157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:29.914388+0000","last_clean_scrub_stamp":"2020-10-29T09:10:18.128157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":"184","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:36.145951+0000","last_change":"2020-10-29T09:10:18.433353+0000","last_active":"2020-10-29T09:10:36.145951+0000","last_peered":"2020-10-29T09:10:36.145951+0000","last_clean":"2020-10-29T09:10:36.145951+0000","last_became_active":"2020-10-29T09:10:18.052992+0000","last_became_peered":"2020-10-29T09:10:18.052992+0000","last_unstale":"2020-10-29T09:10:36.145951+0000","last_undegraded":"2020-10-29T09:10:36.145951+0000","last_fullsized":"2020-10-29T09:10:36.145951+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-10-29T09:10:18.433227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:29.914388+0000","last_clean_scrub_stamp":"2020-10-29T09:10:18.433227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":"178","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:36.064183+0000","last_change":"2020-10-29T09:10:36.062815+0000","last_active":"2020-10-29T09:10:36.064183+0000","last_peered":"2020-10-29T09:10:36.064183+0000","last_clean":"2020-10-29T09:10:36.064183+0000","last_became_active":"2020-10-29T09:09:15.986465+0000","last_became_peered":"2020-10-29T09:09:15.986465+0000","last_unstale":"2020-10-29T09:10:36.064183+0000","last_undegraded":"2020-10-29T09:10:36.064183+0000","last_fullsized":"2020-10-29T09:10:36.064183+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-10-29T09:10:36.062542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:07:05.695099+0000","last_clean_scrub_stamp":"2020-10-29T09:10:36.062542+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":"26","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:36.138011+0000","last_change":"2020-10-29T09:10:19.055939+0000","last_active":"2020-10-29T09:10:36.138011+0000","last_peered":"2020-10-29T09:10:36.138011+0000","last_clean":"2020-10-29T09:10:36.138011+0000","last_became_active":"2020-10-29T09:10:19.055547+0000","last_became_peered":"2020-10-29T09:10:19.055547+0000","last_unstale":"2020-10-29T09:10:36.138011+0000","last_undegraded":"2020-10-29T09:10:36.138011+0000","last_fullsized":"2020-10-29T09:10:36.138011+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-10-29T09:10:14.097420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:09:04.994523+0000","last_clean_scrub_stamp":"2020-10-29T09:10:14.097420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":"189","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:36.065227+0000","last_change":"2020-10-29T09:10:35.103045+0000","last_active":"2020-10-29T09:10:36.065227+0000","last_peered":"2020-10-29T09:10:36.065227+0000","last_clean":"2020-10-29T09:10:36.065227+0000","last_became_active":"2020-10-29T09:10:19.050458+0000","last_became_peered":"2020-10-29T09:10:19.050458+0000","last_unstale":"2020-10-29T09:10:36.065227+0000","last_undegraded":"2020-10-29T09:10:36.065227+0000","last_fullsized":"2020-10-29T09:10:36.065227+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-10-29T09:10:35.102846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:10:35.102846+0000","last_clean_scrub_stamp":"2020-10-29T09:10:35.102846+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":"189","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:35.247196+0000","last_change":"2020-10-29T09:10:06.072332+0000","last_active":"2020-10-29T09:10:35.247196+0000","last_peered":"2020-10-29T09:10:35.247196+0000","last_clean":"2020-10-29T09:10:35.247196+0000","last_became_active":"2020-10-29T09:09:14.992239+0000","last_became_peered":"2020-10-29T09:09:14.992239+0000","last_unstale":"2020-10-29T09:10:35.247196+0000","last_undegraded":"2020-10-29T09:10:35.247196+0000","last_fullsized":"2020-10-29T09:10:35.247196+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-10-29T09:10:06.072137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:29.914388+0000","last_clean_scrub_stamp":"2020-10-29T09:10:06.072137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":"50","reported_epoch":"310","state":"active+clean","last_fresh":"2020-10-29T09:10:39.809320+0000","last_change":"2020-10-29T09:10:39.809527+0000","last_active":"2020-10-29T09:10:39.809320+0000","last_peered":"2020-10-29T09:10:39.809320+0000","last_clean":"2020-10-29T09:10:39.809320+0000","last_became_active":"2020-10-29T09:10:18.051682+0000","last_became_peered":"2020-10-29T09:10:18.051682+0000","last_unstale":"2020-10-29T09:10:39.809320+0000","last_undegraded":"2020-10-29T09:10:39.809320+0000","last_fullsized":"2020-10-29T09:10:39.809320+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-10-29T09:10:39.809319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:29.914388+0000","last_clean_scrub_stamp":"2020-10-29T09:10:39.809319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":"174","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:35.242049+0000","last_change":"2020-10-29T09:09:54.156526+0000","last_active":"2020-10-29T09:10:35.242049+0000","last_peered":"2020-10-29T09:10:35.242049+0000","last_clean":"2020-10-29T09:10:35.242049+0000","last_became_active":"2020-10-29T09:06:56.882000+0000","last_became_peered":"2020-10-29T09:06:56.882000+0000","last_unstale":"2020-10-29T09:10:35.242049+0000","last_undegraded":"2020-10-29T09:10:35.242049+0000","last_fullsized":"2020-10-29T09:10:35.242049+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-10-29T09:09:54.156328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:29.914388+0000","last_clean_scrub_stamp":"2020-10-29T09:09:54.156328+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"213'79","reported_seq":"769","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:35.242258+0000","last_change":"2020-10-29T09:10:15.157748+0000","last_active":"2020-10-29T09:10:35.242258+0000","last_peered":"2020-10-29T09:10:35.242258+0000","last_clean":"2020-10-29T09:10:35.242258+0000","last_became_active":"2020-10-29T09:10:06.933240+0000","last_became_peered":"2020-10-29T09:10:06.933240+0000","last_unstale":"2020-10-29T09:10:35.242258+0000","last_undegraded":"2020-10-29T09:10:35.242258+0000","last_fullsized":"2020-10-29T09:10:35.242258+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'79","last_scrub_stamp":"2020-10-29T09:10:15.157386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:44.778731+0000","last_clean_scrub_stamp":"2020-10-29T09:10:15.157386+0000","log_size":79,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5554176,"num_objects":4,"num_object_clones":3,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":20,"num_read_kb":16277,"num_write":12,"num_write_kb":6132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,7],"acting":[0,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"132","reported_epoch":"310","state":"active+clean","last_fresh":"2020-10-29T09:10:37.189915+0000","last_change":"2020-10-29T09:10:37.190211+0000","last_active":"2020-10-29T09:10:37.189915+0000","last_peered":"2020-10-29T09:10:37.189915+0000","last_clean":"2020-10-29T09:10:37.189915+0000","last_became_active":"2020-10-29T09:10:18.047069+0000","last_became_peered":"2020-10-29T09:10:18.047069+0000","last_unstale":"2020-10-29T09:10:37.189915+0000","last_undegraded":"2020-10-29T09:10:37.189915+0000","last_fullsized":"2020-10-29T09:10:37.189915+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-29T09:10:37.189914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:29.914388+0000","last_clean_scrub_stamp":"2020-10-29T09:10:37.189914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"345","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:35.247343+0000","last_change":"2020-10-29T09:10:24.896603+0000","last_active":"2020-10-29T09:10:35.247343+0000","last_peered":"2020-10-29T09:10:35.247343+0000","last_clean":"2020-10-29T09:10:35.247343+0000","last_became_active":"2020-10-29T09:10:18.064693+0000","last_became_peered":"2020-10-29T09:10:18.064693+0000","last_unstale":"2020-10-29T09:10:35.247343+0000","last_undegraded":"2020-10-29T09:10:35.247343+0000","last_fullsized":"2020-10-29T09:10:35.247343+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-29T09:10:24.896425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:29.914388+0000","last_clean_scrub_stamp":"2020-10-29T09:10:24.896425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"222'333","reported_seq":"1063","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:36.146594+0000","last_change":"2020-10-29T09:10:21.568531+0000","last_active":"2020-10-29T09:10:36.146594+0000","last_peered":"2020-10-29T09:10:36.146594+0000","last_clean":"2020-10-29T09:10:36.146594+0000","last_became_active":"2020-10-29T09:10:10.994104+0000","last_became_peered":"2020-10-29T09:10:10.994104+0000","last_unstale":"2020-10-29T09:10:36.146594+0000","last_undegraded":"2020-10-29T09:10:36.146594+0000","last_fullsized":"2020-10-29T09:10:36.146594+0000","mapping_epoch":288,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'333","last_scrub_stamp":"2020-10-29T09:10:21.568380+0000","last_deep_scrub":"99'124","last_deep_scrub_stamp":"2020-10-29T09:06:03.860758+0000","last_clean_scrub_stamp":"2020-10-29T09:10:21.568380+0000","log_size":333,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17326080,"num_objects":15,"num_object_clones":12,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":2,"num_read":38,"num_read_kb":46781,"num_write":85,"num_write_kb":23340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":50343936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,0],"acting":[3,5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"222'514","reported_seq":"563","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:35.246645+0000","last_change":"2020-10-29T09:10:30.731708+0000","last_active":"2020-10-29T09:10:35.246645+0000","last_peered":"2020-10-29T09:10:35.246645+0000","last_clean":"2020-10-29T09:10:35.246645+0000","last_became_active":"2020-10-29T09:09:19.939411+0000","last_became_peered":"2020-10-29T09:09:19.939411+0000","last_unstale":"2020-10-29T09:10:35.246645+0000","last_undegraded":"2020-10-29T09:10:35.246645+0000","last_fullsized":"2020-10-29T09:10:35.246645+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'514","last_scrub_stamp":"2020-10-29T09:10:30.731485+0000","last_deep_scrub":"222'514","last_deep_scrub_stamp":"2020-10-29T09:10:30.731485+0000","last_clean_scrub_stamp":"2020-10-29T09:10:30.731485+0000","log_size":514,"ondisk_log_size":514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33067008,"num_objects":26,"num_object_clones":21,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":1,"num_read":104,"num_read_kb":94003,"num_write":295,"num_write_kb":34104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":210468864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0,5],"acting":[1,2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"224","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:36.198889+0000","last_change":"2020-10-29T09:10:36.198889+0000","last_active":"2020-10-29T09:10:36.198889+0000","last_peered":"2020-10-29T09:10:36.198889+0000","last_clean":"2020-10-29T09:10:36.198889+0000","last_became_active":"2020-10-29T09:10:35.240183+0000","last_became_peered":"2020-10-29T09:10:35.240183+0000","last_unstale":"2020-10-29T09:10:36.198889+0000","last_undegraded":"2020-10-29T09:10:36.198889+0000","last_fullsized":"2020-10-29T09:10:36.198889+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-29T09:10:36.198717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:29.914388+0000","last_clean_scrub_stamp":"2020-10-29T09:10:36.198717+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"222'97","reported_seq":"852","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:35.241900+0000","last_change":"2020-10-29T09:09:56.088751+0000","last_active":"2020-10-29T09:10:35.241900+0000","last_peered":"2020-10-29T09:10:35.241900+0000","last_clean":"2020-10-29T09:10:35.241900+0000","last_became_active":"2020-10-29T09:09:17.914860+0000","last_became_peered":"2020-10-29T09:09:17.914860+0000","last_unstale":"2020-10-29T09:10:35.241900+0000","last_undegraded":"2020-10-29T09:10:35.241900+0000","last_fullsized":"2020-10-29T09:10:35.241900+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'97","last_scrub_stamp":"2020-10-29T09:09:56.088485+0000","last_deep_scrub":"222'97","last_deep_scrub_stamp":"2020-10-29T09:09:56.088485+0000","last_clean_scrub_stamp":"2020-10-29T09:09:56.088485+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5738496,"num_objects":6,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":21,"num_read_kb":7353,"num_write":23,"num_write_kb":6840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":23912448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3,5],"acting":[0,4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"225","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:35.246393+0000","last_change":"2020-10-29T09:10:35.245612+0000","last_active":"2020-10-29T09:10:35.246393+0000","last_peered":"2020-10-29T09:10:35.246393+0000","last_clean":"2020-10-29T09:10:35.246393+0000","last_became_active":"2020-10-29T09:10:35.244720+0000","last_became_peered":"2020-10-29T09:10:35.244720+0000","last_unstale":"2020-10-29T09:10:35.246393+0000","last_undegraded":"2020-10-29T09:10:35.246393+0000","last_fullsized":"2020-10-29T09:10:35.246393+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-29T09:10:26.878033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:29.914388+0000","last_clean_scrub_stamp":"2020-10-29T09:10:26.878033+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"287'384","reported_seq":"1332","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:36.146368+0000","last_change":"2020-10-29T09:10:22.937378+0000","last_active":"2020-10-29T09:10:36.146368+0000","last_peered":"2020-10-29T09:10:36.146368+0000","last_clean":"2020-10-29T09:10:36.146368+0000","last_became_active":"2020-10-29T09:10:22.936315+0000","last_became_peered":"2020-10-29T09:10:22.936315+0000","last_unstale":"2020-10-29T09:10:36.146368+0000","last_undegraded":"2020-10-29T09:10:36.146368+0000","last_fullsized":"2020-10-29T09:10:36.146368+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'384","last_scrub_stamp":"2020-10-29T09:10:13.495333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:44.778731+0000","last_clean_scrub_stamp":"2020-10-29T09:10:13.495333+0000","log_size":384,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31113216,"num_objects":21,"num_object_clones":17,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":76,"num_read_kb":64013,"num_write":244,"num_write_kb":33300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":157360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6,4],"acting":[3,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'2","reported_seq":"215","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:35.245880+0000","last_change":"2020-10-29T09:10:35.245234+0000","last_active":"2020-10-29T09:10:35.245880+0000","last_peered":"2020-10-29T09:10:35.245880+0000","last_clean":"2020-10-29T09:10:35.245880+0000","last_became_active":"2020-10-29T09:10:35.243577+0000","last_became_peered":"2020-10-29T09:10:35.243577+0000","last_unstale":"2020-10-29T09:10:35.245880+0000","last_undegraded":"2020-10-29T09:10:35.245880+0000","last_fullsized":"2020-10-29T09:10:35.245880+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"16'2","last_scrub_stamp":"2020-10-29T09:10:14.109931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:21.865182+0000","last_clean_scrub_stamp":"2020-10-29T09:10:14.109931+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"326","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:36.064221+0000","last_change":"2020-10-29T09:09:39.035901+0000","last_active":"2020-10-29T09:10:36.064221+0000","last_peered":"2020-10-29T09:10:36.064221+0000","last_clean":"2020-10-29T09:10:36.064221+0000","last_became_active":"2020-10-29T09:09:15.988834+0000","last_became_peered":"2020-10-29T09:09:15.988834+0000","last_unstale":"2020-10-29T09:10:36.064221+0000","last_undegraded":"2020-10-29T09:10:36.064221+0000","last_fullsized":"2020-10-29T09:10:36.064221+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-29T09:09:39.035669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:29.914388+0000","last_clean_scrub_stamp":"2020-10-29T09:09:39.035669+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"295'221","reported_seq":"1091","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:35.246913+0000","last_change":"2020-10-29T09:10:23.931789+0000","last_active":"2020-10-29T09:10:35.246913+0000","last_peered":"2020-10-29T09:10:35.246913+0000","last_clean":"2020-10-29T09:10:35.246913+0000","last_became_active":"2020-10-29T09:10:21.107247+0000","last_became_peered":"2020-10-29T09:10:21.107247+0000","last_unstale":"2020-10-29T09:10:35.246913+0000","last_undegraded":"2020-10-29T09:10:35.246913+0000","last_fullsized":"2020-10-29T09:10:35.246913+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'221","last_scrub_stamp":"2020-10-29T09:10:23.931623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:44.778731+0000","last_clean_scrub_stamp":"2020-10-29T09:10:23.931623+0000","log_size":221,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8380416,"num_objects":11,"num_object_clones":8,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":72,"num_read_kb":45602,"num_write":105,"num_write_kb":11232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":40722432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,5],"acting":[1,2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"143","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:35.241491+0000","last_change":"2020-10-29T09:10:27.170175+0000","last_active":"2020-10-29T09:10:35.241491+0000","last_peered":"2020-10-29T09:10:35.241491+0000","last_clean":"2020-10-29T09:10:35.241491+0000","last_became_active":"2020-10-29T09:10:27.169653+0000","last_became_peered":"2020-10-29T09:10:27.169653+0000","last_unstale":"2020-10-29T09:10:35.241491+0000","last_undegraded":"2020-10-29T09:10:35.241491+0000","last_fullsized":"2020-10-29T09:10:35.241491+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-29T09:09:17.999748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:29.914388+0000","last_clean_scrub_stamp":"2020-10-29T09:09:17.999748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"292'74","reported_seq":"835","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:35.240783+0000","last_change":"2020-10-29T09:10:18.973191+0000","last_active":"2020-10-29T09:10:35.240783+0000","last_peered":"2020-10-29T09:10:35.240783+0000","last_clean":"2020-10-29T09:10:35.240783+0000","last_became_active":"2020-10-29T09:10:18.049215+0000","last_became_peered":"2020-10-29T09:10:18.049215+0000","last_unstale":"2020-10-29T09:10:35.240783+0000","last_undegraded":"2020-10-29T09:10:35.240783+0000","last_fullsized":"2020-10-29T09:10:35.240783+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'74","last_scrub_stamp":"2020-10-29T09:10:18.973056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:44.778731+0000","last_clean_scrub_stamp":"2020-10-29T09:10:18.973056+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11624448,"num_objects":4,"num_object_clones":3,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":14,"num_read_kb":31078,"num_write":29,"num_write_kb":11976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":12754944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,6],"acting":[4,5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"241","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:36.064858+0000","last_change":"2020-10-29T09:10:32.937373+0000","last_active":"2020-10-29T09:10:36.064858+0000","last_peered":"2020-10-29T09:10:36.064858+0000","last_clean":"2020-10-29T09:10:36.064858+0000","last_became_active":"2020-10-29T09:10:32.936749+0000","last_became_peered":"2020-10-29T09:10:32.936749+0000","last_unstale":"2020-10-29T09:10:36.064858+0000","last_undegraded":"2020-10-29T09:10:36.064858+0000","last_fullsized":"2020-10-29T09:10:36.064858+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-29T09:09:29.918286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:29.914388+0000","last_clean_scrub_stamp":"2020-10-29T09:09:29.918286+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"296'738","reported_seq":"1571","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:36.065052+0000","last_change":"2020-10-29T09:10:25.206761+0000","last_active":"2020-10-29T09:10:36.065052+0000","last_peered":"2020-10-29T09:10:36.065052+0000","last_clean":"2020-10-29T09:10:36.065052+0000","last_became_active":"2020-10-29T09:10:21.103170+0000","last_became_peered":"2020-10-29T09:10:21.103170+0000","last_unstale":"2020-10-29T09:10:36.065052+0000","last_undegraded":"2020-10-29T09:10:36.065052+0000","last_fullsized":"2020-10-29T09:10:36.065052+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'738","last_scrub_stamp":"2020-10-29T09:10:25.206541+0000","last_deep_scrub":"296'738","last_deep_scrub_stamp":"2020-10-29T09:10:25.206541+0000","last_clean_scrub_stamp":"2020-10-29T09:10:25.206541+0000","log_size":738,"ondisk_log_size":738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":90968064,"num_objects":38,"num_object_clones":31,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":3,"num_read":196,"num_read_kb":199848,"num_write":290,"num_write_kb":97140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":80044032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2,6],"acting":[5,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"329","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:35.246870+0000","last_change":"2020-10-29T09:10:32.921731+0000","last_active":"2020-10-29T09:10:35.246870+0000","last_peered":"2020-10-29T09:10:35.246870+0000","last_clean":"2020-10-29T09:10:35.246870+0000","last_became_active":"2020-10-29T09:05:28.862975+0000","last_became_peered":"2020-10-29T09:05:28.862975+0000","last_unstale":"2020-10-29T09:10:35.246870+0000","last_undegraded":"2020-10-29T09:10:35.246870+0000","last_fullsized":"2020-10-29T09:10:35.246870+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-29T09:10:32.921569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:29.914388+0000","last_clean_scrub_stamp":"2020-10-29T09:10:32.921569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"295'480","reported_seq":"1040","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:35.241176+0000","last_change":"2020-10-29T09:10:31.199319+0000","last_active":"2020-10-29T09:10:35.241176+0000","last_peered":"2020-10-29T09:10:35.241176+0000","last_clean":"2020-10-29T09:10:35.241176+0000","last_became_active":"2020-10-29T09:10:27.174843+0000","last_became_peered":"2020-10-29T09:10:27.174843+0000","last_unstale":"2020-10-29T09:10:35.241176+0000","last_undegraded":"2020-10-29T09:10:35.241176+0000","last_fullsized":"2020-10-29T09:10:35.241176+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'480","last_scrub_stamp":"2020-10-29T09:10:31.199149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:44.778731+0000","last_clean_scrub_stamp":"2020-10-29T09:10:31.199149+0000","log_size":480,"ondisk_log_size":480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49004544,"num_objects":23,"num_object_clones":19,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":1,"num_read":102,"num_read_kb":138206,"num_write":145,"num_write_kb":48000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":44482560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3,7],"acting":[2,6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"291","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:35.246262+0000","last_change":"2020-10-29T09:10:13.036327+0000","last_active":"2020-10-29T09:10:35.246262+0000","last_peered":"2020-10-29T09:10:35.246262+0000","last_clean":"2020-10-29T09:10:35.246262+0000","last_became_active":"2020-10-29T09:10:06.922006+0000","last_became_peered":"2020-10-29T09:10:06.922006+0000","last_unstale":"2020-10-29T09:10:35.246262+0000","last_undegraded":"2020-10-29T09:10:35.246262+0000","last_fullsized":"2020-10-29T09:10:35.246262+0000","mapping_epoch":284,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-29T09:10:13.036105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:29.914388+0000","last_clean_scrub_stamp":"2020-10-29T09:10:13.036105+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"222'123","reported_seq":"825","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:35.240274+0000","last_change":"2020-10-29T09:10:14.907595+0000","last_active":"2020-10-29T09:10:35.240274+0000","last_peered":"2020-10-29T09:10:35.240274+0000","last_clean":"2020-10-29T09:10:35.240274+0000","last_became_active":"2020-10-29T09:10:09.964579+0000","last_became_peered":"2020-10-29T09:10:09.964579+0000","last_unstale":"2020-10-29T09:10:35.240274+0000","last_undegraded":"2020-10-29T09:10:35.240274+0000","last_fullsized":"2020-10-29T09:10:35.240274+0000","mapping_epoch":287,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'123","last_scrub_stamp":"2020-10-29T09:10:14.907450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:44.778731+0000","last_clean_scrub_stamp":"2020-10-29T09:10:14.907450+0000","log_size":123,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12595200,"num_objects":5,"num_object_clones":4,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":2057,"num_write":79,"num_write_kb":12684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":18849792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7,1],"acting":[4,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"221","reported_epoch":"310","state":"active+clean","last_fresh":"2020-10-29T09:10:37.165809+0000","last_change":"2020-10-29T09:10:37.166004+0000","last_active":"2020-10-29T09:10:37.165809+0000","last_peered":"2020-10-29T09:10:37.165809+0000","last_clean":"2020-10-29T09:10:37.165809+0000","last_became_active":"2020-10-29T09:10:35.244172+0000","last_became_peered":"2020-10-29T09:10:35.244172+0000","last_unstale":"2020-10-29T09:10:37.165809+0000","last_undegraded":"2020-10-29T09:10:37.165809+0000","last_fullsized":"2020-10-29T09:10:37.165809+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-29T09:10:37.165808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:05:41.369125+0000","last_clean_scrub_stamp":"2020-10-29T09:10:37.165808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"296'155","reported_seq":"880","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:36.138032+0000","last_change":"2020-10-29T09:10:33.230436+0000","last_active":"2020-10-29T09:10:36.138032+0000","last_peered":"2020-10-29T09:10:36.138032+0000","last_clean":"2020-10-29T09:10:36.138032+0000","last_became_active":"2020-10-29T09:10:21.106223+0000","last_became_peered":"2020-10-29T09:10:21.106223+0000","last_unstale":"2020-10-29T09:10:36.138032+0000","last_undegraded":"2020-10-29T09:10:36.138032+0000","last_fullsized":"2020-10-29T09:10:36.138032+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'155","last_scrub_stamp":"2020-10-29T09:10:33.230331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:44.778731+0000","last_clean_scrub_stamp":"2020-10-29T09:10:33.230331+0000","log_size":155,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5861376,"num_objects":10,"num_object_clones":8,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":2,"num_read":33,"num_read_kb":39993,"num_write":80,"num_write_kb":7584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3944448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1,7],"acting":[6,4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"195","reported_epoch":"310","state":"active+clean","last_fresh":"2020-10-29T09:10:38.163817+0000","last_change":"2020-10-29T09:10:38.164014+0000","last_active":"2020-10-29T09:10:38.163817+0000","last_peered":"2020-10-29T09:10:38.163817+0000","last_clean":"2020-10-29T09:10:38.163817+0000","last_became_active":"2020-10-29T09:10:35.246399+0000","last_became_peered":"2020-10-29T09:10:35.246399+0000","last_unstale":"2020-10-29T09:10:38.163817+0000","last_undegraded":"2020-10-29T09:10:38.163817+0000","last_fullsized":"2020-10-29T09:10:38.163817+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-29T09:10:38.163816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:10:26.920292+0000","last_clean_scrub_stamp":"2020-10-29T09:10:38.163816+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"289'459","reported_seq":"1154","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:36.146528+0000","last_change":"2020-10-29T09:10:25.605493+0000","last_active":"2020-10-29T09:10:36.146528+0000","last_peered":"2020-10-29T09:10:36.146528+0000","last_clean":"2020-10-29T09:10:36.146528+0000","last_became_active":"2020-10-29T09:10:10.992107+0000","last_became_peered":"2020-10-29T09:10:10.992107+0000","last_unstale":"2020-10-29T09:10:36.146528+0000","last_undegraded":"2020-10-29T09:10:36.146528+0000","last_fullsized":"2020-10-29T09:10:36.146528+0000","mapping_epoch":288,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":289,"parent":"0.0","parent_split_bits":0,"last_scrub":"289'459","last_scrub_stamp":"2020-10-29T09:10:25.605239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:44.778731+0000","last_clean_scrub_stamp":"2020-10-29T09:10:25.605239+0000","log_size":459,"ondisk_log_size":459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32735232,"num_objects":27,"num_object_clones":22,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":2,"num_read":112,"num_read_kb":126486,"num_write":201,"num_write_kb":30072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":8036352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4,7],"acting":[3,0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"325","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:35.246404+0000","last_change":"2020-10-29T09:10:05.104961+0000","last_active":"2020-10-29T09:10:35.246404+0000","last_peered":"2020-10-29T09:10:35.246404+0000","last_clean":"2020-10-29T09:10:35.246404+0000","last_became_active":"2020-10-29T09:05:28.875017+0000","last_became_peered":"2020-10-29T09:05:28.875017+0000","last_unstale":"2020-10-29T09:10:35.246404+0000","last_undegraded":"2020-10-29T09:10:35.246404+0000","last_fullsized":"2020-10-29T09:10:35.246404+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-29T09:10:05.104749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:10:05.104749+0000","last_clean_scrub_stamp":"2020-10-29T09:10:05.104749+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"295'156","reported_seq":"887","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:36.065438+0000","last_change":"2020-10-29T09:10:32.938964+0000","last_active":"2020-10-29T09:10:36.065438+0000","last_peered":"2020-10-29T09:10:36.065438+0000","last_clean":"2020-10-29T09:10:36.065438+0000","last_became_active":"2020-10-29T09:10:32.937785+0000","last_became_peered":"2020-10-29T09:10:32.937785+0000","last_unstale":"2020-10-29T09:10:36.065438+0000","last_undegraded":"2020-10-29T09:10:36.065438+0000","last_fullsized":"2020-10-29T09:10:36.065438+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'156","last_scrub_stamp":"2020-10-29T09:10:24.599546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:44.778731+0000","last_clean_scrub_stamp":"2020-10-29T09:10:24.599546+0000","log_size":156,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8343552,"num_objects":9,"num_object_clones":7,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":10848,"num_write":41,"num_write_kb":9396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1253376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6,1],"acting":[5,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"273","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:35.247192+0000","last_change":"2020-10-29T09:09:52.985794+0000","last_active":"2020-10-29T09:10:35.247192+0000","last_peered":"2020-10-29T09:10:35.247192+0000","last_clean":"2020-10-29T09:10:35.247192+0000","last_became_active":"2020-10-29T09:05:19.989913+0000","last_became_peered":"2020-10-29T09:05:19.989913+0000","last_unstale":"2020-10-29T09:10:35.247192+0000","last_undegraded":"2020-10-29T09:10:35.247192+0000","last_fullsized":"2020-10-29T09:10:35.247192+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-29T09:09:52.985635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:08:29.872856+0000","last_clean_scrub_stamp":"2020-10-29T09:09:52.985635+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"296'590","reported_seq":"370","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:35.242548+0000","last_change":"2020-10-29T09:10:23.375559+0000","last_active":"2020-10-29T09:10:35.242548+0000","last_peered":"2020-10-29T09:10:35.242548+0000","last_clean":"2020-10-29T09:10:35.242548+0000","last_became_active":"2020-10-29T09:10:21.108911+0000","last_became_peered":"2020-10-29T09:10:21.108911+0000","last_unstale":"2020-10-29T09:10:35.242548+0000","last_undegraded":"2020-10-29T09:10:35.242548+0000","last_fullsized":"2020-10-29T09:10:35.242548+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'590","last_scrub_stamp":"2020-10-29T09:10:23.375353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:44.778731+0000","last_clean_scrub_stamp":"2020-10-29T09:10:23.375353+0000","log_size":590,"ondisk_log_size":590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34258944,"num_objects":31,"num_object_clones":26,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":2,"num_read":121,"num_read_kb":74298,"num_write":247,"num_write_kb":31896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":103120896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1,6],"acting":[0,5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":"187","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:35.241853+0000","last_change":"2020-10-29T09:09:30.962930+0000","last_active":"2020-10-29T09:10:35.241853+0000","last_peered":"2020-10-29T09:10:35.241853+0000","last_clean":"2020-10-29T09:10:35.241853+0000","last_became_active":"2020-10-29T09:09:14.998147+0000","last_became_peered":"2020-10-29T09:09:14.998147+0000","last_unstale":"2020-10-29T09:10:35.241853+0000","last_undegraded":"2020-10-29T09:10:35.241853+0000","last_fullsized":"2020-10-29T09:10:35.241853+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-29T09:09:30.962849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:29.914388+0000","last_clean_scrub_stamp":"2020-10-29T09:09:30.962849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"285'148","reported_seq":"815","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:35.248894+0000","last_change":"2020-10-29T09:10:35.248194+0000","last_active":"2020-10-29T09:10:35.248894+0000","last_peered":"2020-10-29T09:10:35.248894+0000","last_clean":"2020-10-29T09:10:35.248894+0000","last_became_active":"2020-10-29T09:10:35.246716+0000","last_became_peered":"2020-10-29T09:10:35.246716+0000","last_unstale":"2020-10-29T09:10:35.248894+0000","last_undegraded":"2020-10-29T09:10:35.248894+0000","last_fullsized":"2020-10-29T09:10:35.248894+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'148","last_scrub_stamp":"2020-10-29T09:10:12.918856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:44.778731+0000","last_clean_scrub_stamp":"2020-10-29T09:10:12.918856+0000","log_size":148,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5922816,"num_objects":6,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":25,"num_read_kb":25099,"num_write":103,"num_write_kb":5784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,2],"acting":[7,4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":"276","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:35.240932+0000","last_change":"2020-10-29T09:10:22.409902+0000","last_active":"2020-10-29T09:10:35.240932+0000","last_peered":"2020-10-29T09:10:35.240932+0000","last_clean":"2020-10-29T09:10:35.240932+0000","last_became_active":"2020-10-29T09:05:19.982705+0000","last_became_peered":"2020-10-29T09:05:19.982705+0000","last_unstale":"2020-10-29T09:10:35.240932+0000","last_undegraded":"2020-10-29T09:10:35.240932+0000","last_fullsized":"2020-10-29T09:10:35.240932+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-29T09:10:22.409519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:29.914388+0000","last_clean_scrub_stamp":"2020-10-29T09:10:22.409519+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"289'589","reported_seq":"871","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:35.249886+0000","last_change":"2020-10-29T09:10:35.249051+0000","last_active":"2020-10-29T09:10:35.249886+0000","last_peered":"2020-10-29T09:10:35.249886+0000","last_clean":"2020-10-29T09:10:35.249886+0000","last_became_active":"2020-10-29T09:10:35.246732+0000","last_became_peered":"2020-10-29T09:10:35.246732+0000","last_unstale":"2020-10-29T09:10:35.249886+0000","last_undegraded":"2020-10-29T09:10:35.249886+0000","last_fullsized":"2020-10-29T09:10:35.249886+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"289'589","last_scrub_stamp":"2020-10-29T09:10:13.917602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:44.778731+0000","last_clean_scrub_stamp":"2020-10-29T09:10:13.917602+0000","log_size":589,"ondisk_log_size":589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36593664,"num_objects":27,"num_object_clones":22,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":1,"num_read":101,"num_read_kb":117242,"num_write":247,"num_write_kb":43248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":11243520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0,1],"acting":[7,4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":"93","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:36.137895+0000","last_change":"2020-10-29T09:10:34.186980+0000","last_active":"2020-10-29T09:10:36.137895+0000","last_peered":"2020-10-29T09:10:36.137895+0000","last_clean":"2020-10-29T09:10:36.137895+0000","last_became_active":"2020-10-29T09:10:18.054985+0000","last_became_peered":"2020-10-29T09:10:18.054985+0000","last_unstale":"2020-10-29T09:10:36.137895+0000","last_undegraded":"2020-10-29T09:10:36.137895+0000","last_fullsized":"2020-10-29T09:10:36.137895+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-29T09:10:34.186781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:29.914388+0000","last_clean_scrub_stamp":"2020-10-29T09:10:34.186781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":"100","reported_epoch":"310","state":"active+clean","last_fresh":"2020-10-29T09:10:36.965300+0000","last_change":"2020-10-29T09:10:36.965443+0000","last_active":"2020-10-29T09:10:36.965300+0000","last_peered":"2020-10-29T09:10:36.965300+0000","last_clean":"2020-10-29T09:10:36.965300+0000","last_became_active":"2020-10-29T09:10:27.861518+0000","last_became_peered":"2020-10-29T09:10:27.861518+0000","last_unstale":"2020-10-29T09:10:36.965300+0000","last_undegraded":"2020-10-29T09:10:36.965300+0000","last_fullsized":"2020-10-29T09:10:36.965300+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-10-29T09:10:36.965299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:29.914388+0000","last_clean_scrub_stamp":"2020-10-29T09:10:36.965299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":"129","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:35.241244+0000","last_change":"2020-10-29T09:10:27.171362+0000","last_active":"2020-10-29T09:10:35.241244+0000","last_peered":"2020-10-29T09:10:35.241244+0000","last_clean":"2020-10-29T09:10:35.241244+0000","last_became_active":"2020-10-29T09:10:27.170878+0000","last_became_peered":"2020-10-29T09:10:27.170878+0000","last_unstale":"2020-10-29T09:10:35.241244+0000","last_undegraded":"2020-10-29T09:10:35.241244+0000","last_fullsized":"2020-10-29T09:10:35.241244+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-10-29T09:09:50.775502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:29.914388+0000","last_clean_scrub_stamp":"2020-10-29T09:09:50.775502+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":"352","reported_epoch":"309","state":"active+clean","last_fresh":"2020-10-29T09:10:36.064415+0000","last_change":"2020-10-29T09:09:52.162831+0000","last_active":"2020-10-29T09:10:36.064415+0000","last_peered":"2020-10-29T09:10:36.064415+0000","last_clean":"2020-10-29T09:10:36.064415+0000","last_became_active":"2020-10-29T09:09:15.990392+0000","last_became_peered":"2020-10-29T09:09:15.990392+0000","last_unstale":"2020-10-29T09:10:36.064415+0000","last_undegraded":"2020-10-29T09:10:36.064415+0000","last_fullsized":"2020-10-29T09:10:36.064415+0000","mapping_epoch":259,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-10-29T09:09:52.162641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-29T09:04:29.914388+0000","last_clean_scrub_stamp":"2020-10-29T09:09:52.162641+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":389087232,"num_objects":263,"num_object_clones":213,"num_object_copies":1052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":263,"num_whiteouts":16,"num_read":1077,"num_read_kb":1039184,"num_write":2226,"num_write_kb":412728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":549,"num_bytes_recovered":766537728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":560607232,"data_stored":908754944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23935,"internal_metadata":0},"log_size":5140,"ondisk_log_size":5140,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13873,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":28,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":56,"acting":56,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":284,"seq":1219770712073,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94932,"kb_used_data":63308,"kb_used_omap":166,"kb_used_meta":31449,"kb_avail":94276908,"statfs":{"total":96636764160,"available":96539553792,"internally_reserved":0,"allocated":64827392,"data_stored":115347190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":170905,"internal_metadata":32203879},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.097}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.536}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.702}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3079999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3719999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1890000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2149999999999999}]}]},{"osd":6,"up_from":291,"seq":1249835483143,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":142336,"kb_used_data":106936,"kb_used_omap":51,"kb_used_meta":35340,"kb_avail":94229504,"statfs":{"total":96636764160,"available":96491012096,"internally_reserved":0,"allocated":109502464,"data_stored":131415798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52355,"internal_metadata":36189053},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2669999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1909999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3570000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6960000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4990000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0870000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9780000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542222,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":158444,"kb_used_data":66724,"kb_used_omap":0,"kb_used_meta":91712,"kb_avail":94213396,"statfs":{"total":96636764160,"available":96474517504,"internally_reserved":0,"allocated":68325376,"data_stored":92978934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":93913088},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 29 09:10:35 2020","interfaces":[{"interface":"back","average":{"1min":2.2160000000000002,"5min":2.0230000000000001,"15min":39.938000000000002},"min":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"max":{"1min":3.7400000000000002,"5min":5.4340000000000002,"15min":991.51300000000003},"last":2.0899999999999999},{"interface":"front","average":{"1min":2.0539999999999998,"5min":1.9510000000000001,"15min":2.0699999999999998},"min":{"1min":1.1619999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":4.0410000000000004,"5min":4.0410000000000004,"15min":4.0410000000000004},"last":2.2480000000000002}]},{"osd":2,"last update":"Thu Oct 29 09:10:35 2020","interfaces":[{"interface":"back","average":{"1min":2.0019999999999998,"5min":2.0230000000000001,"15min":2.0459999999999998},"min":{"1min":1.131,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":3.5099999999999998,"5min":4.9729999999999999,"15min":4.9729999999999999},"last":2.157},{"interface":"front","average":{"1min":47.127000000000002,"5min":18.369,"15min":7.649},"min":{"1min":1.3149999999999999,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":803.279,"5min":803.279,"15min":803.279},"last":2.879}]},{"osd":3,"last update":"Thu Oct 29 09:10:35 2020","interfaces":[{"interface":"back","average":{"1min":2.8530000000000002,"5min":2.5510000000000002,"15min":21.643999999999998},"min":{"1min":1.6830000000000001,"5min":1.0669999999999999,"15min":1.0669999999999999},"max":{"1min":4.7480000000000002,"5min":5.1849999999999996,"15min":752.56299999999999},"last":2.9729999999999999},{"interface":"front","average":{"1min":3.004,"5min":2.3849999999999998,"15min":2.3010000000000002},"min":{"1min":1.7769999999999999,"5min":1.0780000000000001,"15min":1.0780000000000001},"max":{"1min":4.8470000000000004,"5min":4.8470000000000004,"15min":4.8470000000000004},"last":3.0299999999999998}]},{"osd":4,"last update":"Thu Oct 29 09:10:35 2020","interfaces":[{"interface":"back","average":{"1min":2.2050000000000001,"5min":2.1469999999999998,"15min":14.952999999999999},"min":{"1min":1.256,"5min":1.1870000000000001,"15min":1.1870000000000001},"max":{"1min":3.238,"5min":5.1849999999999996,"15min":494.577},"last":2.5939999999999999},{"interface":"front","average":{"1min":25.248999999999999,"5min":7.0250000000000004,"15min":4.3129999999999997},"min":{"1min":1.5009999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"max":{"1min":412.005,"5min":412.005,"15min":412.005},"last":2.802}]},{"osd":5,"last update":"Thu Oct 29 09:10:36 2020","interfaces":[{"interface":"back","average":{"1min":2.6320000000000001,"5min":5.6520000000000001,"15min":3.9169999999999998},"min":{"1min":1.7190000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"max":{"1min":3.7410000000000001,"5min":305.01600000000002,"15min":305.01600000000002},"last":2.931},{"interface":"front","average":{"1min":3.0739999999999998,"5min":4.5679999999999996,"15min":3.4500000000000002},"min":{"1min":1.98,"5min":1.1539999999999999,"15min":1.1539999999999999},"max":{"1min":4.96,"5min":187.482,"15min":187.482},"last":3.2530000000000001}]},{"osd":6,"last update":"Thu Oct 29 09:06:33 2020","interfaces":[{"interface":"back","average":{"1min":2.7589999999999999,"5min":2.8330000000000002,"15min":2.8450000000000002},"min":{"1min":1.675,"5min":1.5489999999999999,"15min":1.5489999999999999},"max":{"1min":3.907,"5min":4.9880000000000004,"15min":4.9880000000000004},"last":4.2910000000000004},{"interface":"front","average":{"1min":2.9900000000000002,"5min":5.6980000000000004,"15min":6.149},"min":{"1min":1.708,"5min":1.708,"15min":1.708},"max":{"1min":3.9199999999999999,"5min":80.652000000000001,"15min":80.652000000000001},"last":3.7490000000000001}]},{"osd":7,"last update":"Thu Oct 29 09:07:29 2020","interfaces":[{"interface":"back","average":{"1min":3.036,"5min":3.0939999999999999,"15min":3.1150000000000002},"min":{"1min":1.782,"5min":1.194,"15min":1.194},"max":{"1min":4.0469999999999997,"5min":5.7110000000000003,"15min":5.7110000000000003},"last":4.1299999999999999},{"interface":"front","average":{"1min":3.1659999999999999,"5min":3.024,"15min":3.048},"min":{"1min":1.889,"5min":1.3240000000000001,"15min":1.3240000000000001},"max":{"1min":4.1139999999999999,"5min":5.0730000000000004,"15min":5.0730000000000004},"last":3.8410000000000002}]}]},{"osd":0,"up_from":14,"seq":60129542222,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":147552,"kb_used_data":68504,"kb_used_omap":0,"kb_used_meta":79040,"kb_avail":94224288,"statfs":{"total":96636764160,"available":96485670912,"internally_reserved":0,"allocated":70148096,"data_stored":103149302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":80936960},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Oct 29 09:09:38 2020","interfaces":[{"interface":"back","average":{"1min":1.77,"5min":12.827999999999999,"15min":5.9569999999999999},"min":{"1min":1.032,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":2.7719999999999998,"5min":521.721,"15min":521.721},"last":3.9849999999999999},{"interface":"front","average":{"1min":1.901,"5min":11.836,"15min":5.7809999999999997},"min":{"1min":1.1419999999999999,"5min":0.90000000000000002,"15min":0.90000000000000002},"max":{"1min":2.7770000000000001,"5min":504.34800000000001,"15min":504.34800000000001},"last":4.2229999999999999}]},{"osd":2,"last update":"Thu Oct 29 09:09:38 2020","interfaces":[{"interface":"back","average":{"1min":1.931,"5min":11.173,"15min":5.0439999999999996},"min":{"1min":1.2190000000000001,"5min":1.016,"15min":1.016},"max":{"1min":3.504,"5min":875.03300000000002,"15min":875.03300000000002},"last":3.0840000000000001},{"interface":"front","average":{"1min":33.969000000000001,"5min":8.5340000000000007,"15min":4.2990000000000004},"min":{"1min":1.3580000000000001,"5min":0.98499999999999999,"15min":0.98499999999999999},"max":{"1min":639.08799999999997,"5min":639.08799999999997,"15min":639.08799999999997},"last":3.3919999999999999}]},{"osd":3,"last update":"Thu Oct 29 09:09:38 2020","interfaces":[{"interface":"back","average":{"1min":2.2650000000000001,"5min":2.5529999999999999,"15min":2.839},"min":{"1min":1.675,"5min":1.3280000000000001,"15min":1.3280000000000001},"max":{"1min":2.8860000000000001,"5min":14.515000000000001,"15min":14.515000000000001},"last":4.0970000000000004},{"interface":"front","average":{"1min":2.101,"5min":2.302,"15min":2.5190000000000001},"min":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"max":{"1min":3.0219999999999998,"5min":8.4589999999999996,"15min":8.4589999999999996},"last":2.9049999999999998}]},{"osd":4,"last update":"Thu Oct 29 09:09:38 2020","interfaces":[{"interface":"back","average":{"1min":2.238,"5min":2.3599999999999999,"15min":2.4700000000000002},"min":{"1min":1.6739999999999999,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":4.1159999999999997,"5min":6.5430000000000001,"15min":6.5430000000000001},"last":3.7599999999999998},{"interface":"front","average":{"1min":2.6970000000000001,"5min":3.0009999999999999,"15min":3.3530000000000002},"min":{"1min":1.8819999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"max":{"1min":3.258,"5min":14.443,"15min":14.443},"last":39.201000000000001}]},{"osd":5,"last update":"Thu Oct 29 09:09:38 2020","interfaces":[{"interface":"back","average":{"1min":2.698,"5min":3.101,"15min":3.3809999999999998},"min":{"1min":1.8839999999999999,"5min":1.45,"15min":1.45},"max":{"1min":3.0910000000000002,"5min":13.707000000000001,"15min":13.707000000000001},"last":4.6239999999999997},{"interface":"front","average":{"1min":2.4209999999999998,"5min":15.853,"15min":7.1559999999999997},"min":{"1min":1.964,"5min":1.478,"15min":1.478},"max":{"1min":3.274,"5min":993.60000000000002,"15min":993.60000000000002},"last":3.1749999999999998}]},{"osd":6,"last update":"Thu Oct 29 09:07:32 2020","interfaces":[{"interface":"back","average":{"1min":2.843,"5min":2.8610000000000002,"15min":2.8929999999999998},"min":{"1min":1.954,"5min":1.671,"15min":1.671},"max":{"1min":3.4470000000000001,"5min":4.6879999999999997,"15min":4.6879999999999997},"last":5.6120000000000001},{"interface":"front","average":{"1min":2.9449999999999998,"5min":2.9980000000000002,"15min":3.1019999999999999},"min":{"1min":1.8819999999999999,"5min":1.623,"15min":1.623},"max":{"1min":3.9369999999999998,"5min":6.6239999999999997,"15min":6.6239999999999997},"last":4.2999999999999998}]},{"osd":7,"last update":"Thu Oct 29 09:07:32 2020","interfaces":[{"interface":"back","average":{"1min":2.9940000000000002,"5min":18.454000000000001,"15min":25.379000000000001},"min":{"1min":2.177,"5min":1.3819999999999999,"15min":1.3819999999999999},"max":{"1min":3.8599999999999999,"5min":565.779,"15min":565.779},"last":3.6819999999999999},{"interface":"front","average":{"1min":30.641999999999999,"5min":8.9440000000000008,"15min":5.4470000000000001},"min":{"1min":2.3359999999999999,"5min":1.25,"15min":1.25},"max":{"1min":522.76300000000003,"5min":522.76300000000003,"15min":522.76300000000003},"last":3.4710000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542223,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":136684,"kb_used_data":68836,"kb_used_omap":0,"kb_used_meta":67840,"kb_avail":94235156,"statfs":{"total":96636764160,"available":96496799744,"internally_reserved":0,"allocated":70488064,"data_stored":119512822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":69468160},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 29 09:10:38 2020","interfaces":[{"interface":"back","average":{"1min":1.863,"5min":8.1430000000000007,"15min":19.914999999999999},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":3.1970000000000001,"5min":427.91500000000002,"15min":502.91399999999999},"last":1.5329999999999999},{"interface":"front","average":{"1min":1.782,"5min":1.9179999999999999,"15min":32.496000000000002},"min":{"1min":0.96499999999999997,"5min":0.92400000000000004,"15min":0.76700000000000002},"max":{"1min":2.782,"5min":3.9430000000000001,"15min":502.99799999999999},"last":1.6020000000000001}]},{"osd":1,"last update":"Thu Oct 29 09:10:38 2020","interfaces":[{"interface":"back","average":{"1min":1.841,"5min":12.340999999999999,"15min":5.5350000000000001},"min":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"max":{"1min":2.5449999999999999,"5min":974.00099999999998,"15min":974.00099999999998},"last":2.1139999999999999},{"interface":"front","average":{"1min":32.972999999999999,"5min":28.324999999999999,"15min":11.151999999999999},"min":{"1min":0.96299999999999997,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":558.37900000000002,"5min":947.11199999999997,"15min":947.11199999999997},"last":2.0139999999999998}]},{"osd":3,"last update":"Thu Oct 29 09:10:38 2020","interfaces":[{"interface":"back","average":{"1min":26.829999999999998,"5min":7.3550000000000004,"15min":3.9430000000000001},"min":{"1min":1.6719999999999999,"5min":1.117,"15min":1.117},"max":{"1min":464.81999999999999,"5min":464.81999999999999,"15min":464.81999999999999},"last":464.81999999999999},{"interface":"front","average":{"1min":2.4380000000000002,"5min":12.039,"15min":5.7050000000000001},"min":{"1min":1.5609999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"max":{"1min":4.508,"5min":955.24000000000001,"15min":955.24000000000001},"last":1.7909999999999999}]},{"osd":4,"last update":"Thu Oct 29 09:10:38 2020","interfaces":[{"interface":"back","average":{"1min":2.2480000000000002,"5min":18.745999999999999,"15min":36.359999999999999},"min":{"1min":1.306,"5min":1.2250000000000001,"15min":1.2250000000000001},"max":{"1min":4.141,"5min":504.06,"15min":853.03899999999999},"last":1.869},{"interface":"front","average":{"1min":2.427,"5min":2.5800000000000001,"15min":45.173999999999999},"min":{"1min":1.4950000000000001,"5min":1.145,"15min":1.145},"max":{"1min":4.069,"5min":4.5119999999999996,"15min":942.76700000000005},"last":2.3639999999999999}]},{"osd":5,"last update":"Thu Oct 29 09:10:38 2020","interfaces":[{"interface":"back","average":{"1min":2.3479999999999999,"5min":2.383,"15min":33.274999999999999},"min":{"1min":1.2669999999999999,"5min":0.98299999999999998,"15min":0.98299999999999998},"max":{"1min":4.3259999999999996,"5min":4.3259999999999996,"15min":503.68000000000001},"last":2.1819999999999999},{"interface":"front","average":{"1min":2.7109999999999999,"5min":2.8639999999999999,"15min":2.8999999999999999},"min":{"1min":1.821,"5min":1.446,"15min":1.446},"max":{"1min":4.0060000000000002,"5min":4.4820000000000002,"15min":4.7460000000000004},"last":2.7719999999999998}]},{"osd":6,"last update":"Thu Oct 29 09:07:32 2020","interfaces":[{"interface":"back","average":{"1min":2.8879999999999999,"5min":3.1539999999999999,"15min":3.238},"min":{"1min":1.252,"5min":1.252,"15min":1.252},"max":{"1min":4.0640000000000001,"5min":7.3440000000000003,"15min":7.3440000000000003},"last":3.0920000000000001},{"interface":"front","average":{"1min":2.6190000000000002,"5min":2.758,"15min":2.855},"min":{"1min":1.397,"5min":1.397,"15min":1.397},"max":{"1min":3.7530000000000001,"5min":6.1040000000000001,"15min":6.1040000000000001},"last":3.0609999999999999}]},{"osd":7,"last update":"Thu Oct 29 09:07:32 2020","interfaces":[{"interface":"back","average":{"1min":3.1499999999999999,"5min":10.390000000000001,"15min":5.7350000000000003},"min":{"1min":1.597,"5min":1.597,"15min":1.597},"max":{"1min":4.5910000000000002,"5min":709.851,"15min":709.851},"last":2.7229999999999999},{"interface":"front","average":{"1min":41.984000000000002,"5min":10.823,"15min":5.6719999999999997},"min":{"1min":1.857,"5min":1.8129999999999999,"15min":1.8129999999999999},"max":{"1min":700.84100000000001,"5min":700.84100000000001,"15min":700.84100000000001},"last":2.855}]}]},{"osd":3,"up_from":14,"seq":60129542222,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":143356,"kb_used_data":70260,"kb_used_omap":0,"kb_used_meta":73088,"kb_avail":94228484,"statfs":{"total":96636764160,"available":96489967616,"internally_reserved":0,"allocated":71946240,"data_stored":138493686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":74842112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 29 09:09:37 2020","interfaces":[{"interface":"back","average":{"1min":1.9179999999999999,"5min":7.5579999999999998,"15min":19.989000000000001},"min":{"1min":1.2210000000000001,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":2.411,"5min":503.108,"15min":503.108},"last":1.512},{"interface":"front","average":{"1min":2.2309999999999999,"5min":9.8290000000000006,"15min":30.925000000000001},"min":{"1min":1.3340000000000001,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":3.4249999999999998,"5min":502.49099999999999,"15min":502.49099999999999},"last":1.7470000000000001}]},{"osd":1,"last update":"Thu Oct 29 09:09:37 2020","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.823,"15min":1.867},"min":{"1min":0.89200000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":2.9529999999999998,"5min":3.8170000000000002,"15min":3.8170000000000002},"last":2.8570000000000002},{"interface":"front","average":{"1min":1.966,"5min":22.745000000000001,"15min":8.9749999999999996},"min":{"1min":1.3200000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"max":{"1min":2.581,"5min":909.08000000000004,"15min":909.08000000000004},"last":1.1519999999999999}]},{"osd":2,"last update":"Thu Oct 29 09:09:37 2020","interfaces":[{"interface":"back","average":{"1min":2.0720000000000001,"5min":6.9029999999999996,"15min":19.934999999999999},"min":{"1min":1.52,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":2.8479999999999999,"5min":516.42200000000003,"15min":516.42200000000003},"last":2.7549999999999999},{"interface":"front","average":{"1min":2.1440000000000001,"5min":4.9370000000000003,"15min":3.2570000000000001},"min":{"1min":1.4059999999999999,"5min":0.97199999999999998,"15min":0.97199999999999998},"max":{"1min":2.8380000000000001,"5min":182.91499999999999,"15min":182.91499999999999},"last":1.843}]},{"osd":4,"last update":"Thu Oct 29 09:09:37 2020","interfaces":[{"interface":"back","average":{"1min":2.2200000000000002,"5min":2.5870000000000002,"15min":2.6669999999999998},"min":{"1min":1.518,"5min":1.339,"15min":1.339},"max":{"1min":3.157,"5min":7.0190000000000001,"15min":7.0190000000000001},"last":1.454},{"interface":"front","average":{"1min":2.6850000000000001,"5min":11.487,"15min":5.4870000000000001},"min":{"1min":2.069,"5min":1.325,"15min":1.325},"max":{"1min":3.4670000000000001,"5min":619.63099999999997,"15min":619.63099999999997},"last":2.9830000000000001}]},{"osd":5,"last update":"Thu Oct 29 09:09:37 2020","interfaces":[{"interface":"back","average":{"1min":2.8620000000000001,"5min":12.853,"15min":6.0910000000000002},"min":{"1min":1.9590000000000001,"5min":1.1319999999999999,"15min":1.1319999999999999},"max":{"1min":3.54,"5min":427.04399999999998,"15min":427.04399999999998},"last":3.137},{"interface":"front","average":{"1min":2.8769999999999998,"5min":10.194000000000001,"15min":29.616},"min":{"1min":1.9339999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"max":{"1min":3.6299999999999999,"5min":741.56500000000005,"15min":741.56500000000005},"last":3.2029999999999998}]},{"osd":6,"last update":"Thu Oct 29 09:07:32 2020","interfaces":[{"interface":"back","average":{"1min":2.4399999999999999,"5min":2.7879999999999998,"15min":2.7810000000000001},"min":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"max":{"1min":3.3570000000000002,"5min":5.5979999999999999,"15min":5.5979999999999999},"last":2.927},{"interface":"front","average":{"1min":2.9199999999999999,"5min":2.9710000000000001,"15min":2.9510000000000001},"min":{"1min":2.0659999999999998,"5min":1.762,"15min":1.762},"max":{"1min":3.8610000000000002,"5min":5.9790000000000001,"15min":5.9790000000000001},"last":3.044}]},{"osd":7,"last update":"Thu Oct 29 09:07:32 2020","interfaces":[{"interface":"back","average":{"1min":3.4359999999999999,"5min":11.807,"15min":6.1589999999999998},"min":{"1min":1.669,"5min":1.482,"15min":1.482},"max":{"1min":4.6719999999999997,"5min":545.26099999999997,"15min":545.26099999999997},"last":3.5099999999999998},{"interface":"front","average":{"1min":3.5219999999999998,"5min":3.3340000000000001,"15min":3.1539999999999999},"min":{"1min":1.6919999999999999,"5min":1.661,"15min":1.661},"max":{"1min":4.7270000000000003,"5min":7.0519999999999996,"15min":7.0519999999999996},"last":3.4790000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542222,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":153080,"kb_used_data":48048,"kb_used_omap":0,"kb_used_meta":105024,"kb_avail":94218760,"statfs":{"total":96636764160,"available":96480010240,"internally_reserved":0,"allocated":49201152,"data_stored":95928054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":107544576},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 29 09:10:34 2020","interfaces":[{"interface":"back","average":{"1min":16.672000000000001,"5min":14.718,"15min":6.3730000000000002},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"max":{"1min":316.39100000000002,"5min":503.32999999999998,"15min":503.32999999999998},"last":2.2050000000000001},{"interface":"front","average":{"1min":1.8160000000000001,"5min":24.957000000000001,"15min":9.7390000000000008},"min":{"1min":1.0900000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":3.1720000000000002,"5min":791.57399999999996,"15min":791.57399999999996},"last":2.681}]},{"osd":1,"last update":"Thu Oct 29 09:10:34 2020","interfaces":[{"interface":"back","average":{"1min":2.0579999999999998,"5min":1.9930000000000001,"15min":2.2229999999999999},"min":{"1min":1.4059999999999999,"5min":1.0760000000000001,"15min":1.0309999999999999},"max":{"1min":3.0009999999999999,"5min":5.1150000000000002,"15min":5.1150000000000002},"last":2.3100000000000001},{"interface":"front","average":{"1min":2.1560000000000001,"5min":2.1960000000000002,"15min":2.367},"min":{"1min":1.385,"5min":1.0669999999999999,"15min":1.0669999999999999},"max":{"1min":3.3610000000000002,"5min":6.4690000000000003,"15min":6.4690000000000003},"last":2.4620000000000002}]},{"osd":2,"last update":"Thu Oct 29 09:10:34 2020","interfaces":[{"interface":"back","average":{"1min":2.282,"5min":6.5819999999999999,"15min":3.8780000000000001},"min":{"1min":1.345,"5min":1.194,"15min":1.194},"max":{"1min":3.4729999999999999,"5min":346.06599999999997,"15min":346.06599999999997},"last":3.0249999999999999},{"interface":"front","average":{"1min":2.1920000000000002,"5min":2.3359999999999999,"15min":2.605},"min":{"1min":1.3,"5min":1.3,"15min":1.3},"max":{"1min":3.383,"5min":6.5289999999999999,"15min":6.5289999999999999},"last":2.9079999999999999}]},{"osd":3,"last update":"Thu Oct 29 09:10:34 2020","interfaces":[{"interface":"back","average":{"1min":19.280000000000001,"5min":5.8760000000000003,"15min":3.8410000000000002},"min":{"1min":1.7030000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"max":{"1min":357.673,"5min":357.673,"15min":357.673},"last":2.5489999999999999},{"interface":"front","average":{"1min":2.3090000000000002,"5min":2.3889999999999998,"15min":2.6779999999999999},"min":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"max":{"1min":3.4239999999999999,"5min":6.2510000000000003,"15min":6.2510000000000003},"last":2.7690000000000001}]},{"osd":5,"last update":"Thu Oct 29 09:10:34 2020","interfaces":[{"interface":"back","average":{"1min":2.4390000000000001,"5min":9.4770000000000003,"15min":18.289999999999999},"min":{"1min":1.925,"5min":0.91300000000000003,"15min":0.91300000000000003},"max":{"1min":3.6749999999999998,"5min":484.32600000000002,"15min":484.32600000000002},"last":3.1499999999999999},{"interface":"front","average":{"1min":2.5899999999999999,"5min":6.8849999999999998,"15min":4.3109999999999999},"min":{"1min":1.7130000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"max":{"1min":3.6240000000000001,"5min":365.31400000000002,"15min":365.31400000000002},"last":2.601}]},{"osd":6,"last update":"Thu Oct 29 09:07:31 2020","interfaces":[{"interface":"back","average":{"1min":3.2269999999999999,"5min":4.4249999999999998,"15min":3.6419999999999999},"min":{"1min":2.3799999999999999,"5min":1.625,"15min":1.625},"max":{"1min":6.2779999999999996,"5min":160.38300000000001,"15min":160.38300000000001},"last":2.383},{"interface":"front","average":{"1min":2.8450000000000002,"5min":2.9969999999999999,"15min":3.0539999999999998},"min":{"1min":2.004,"5min":1.1799999999999999,"15min":1.1799999999999999},"max":{"1min":3.7629999999999999,"5min":5.1890000000000001,"15min":5.1890000000000001},"last":2.851}]},{"osd":7,"last update":"Thu Oct 29 09:07:31 2020","interfaces":[{"interface":"back","average":{"1min":3.3839999999999999,"5min":3.5070000000000001,"15min":3.609},"min":{"1min":2.3239999999999998,"5min":1.446,"15min":1.446},"max":{"1min":6.609,"5min":10.282,"15min":10.282},"last":3.2709999999999999},{"interface":"front","average":{"1min":20.803999999999998,"5min":6.7549999999999999,"15min":4.444},"min":{"1min":2.7530000000000001,"5min":1.748,"15min":1.748},"max":{"1min":367.77100000000002,"5min":367.77100000000002,"15min":367.77100000000002},"last":3.0859999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542222,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":174764,"kb_used_data":84260,"kb_used_omap":0,"kb_used_meta":90496,"kb_avail":94197076,"statfs":{"total":96636764160,"available":96457805824,"internally_reserved":0,"allocated":86282240,"data_stored":128278262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":92667904},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Oct 29 09:10:34 2020","interfaces":[{"interface":"back","average":{"1min":1.6459999999999999,"5min":1.6639999999999999,"15min":2.032},"min":{"1min":1.0800000000000001,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":2.9870000000000001,"5min":3.0129999999999999,"15min":3.431},"last":1.3069999999999999},{"interface":"front","average":{"1min":2.0409999999999999,"5min":11.782,"15min":5.2789999999999999},"min":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"max":{"1min":3.2200000000000002,"5min":843.72000000000003,"15min":843.72000000000003},"last":2.8479999999999999}]},{"osd":1,"last update":"Thu Oct 29 09:10:34 2020","interfaces":[{"interface":"back","average":{"1min":1.966,"5min":8.3569999999999993,"15min":4.3369999999999997},"min":{"1min":1.1499999999999999,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":3.4830000000000001,"5min":625.16099999999994,"15min":625.16099999999994},"last":2.3319999999999999},{"interface":"front","average":{"1min":2.3500000000000001,"5min":5.5049999999999999,"15min":3.4209999999999998},"min":{"1min":1.5960000000000001,"5min":0.95299999999999996,"15min":0.95299999999999996},"max":{"1min":3.831,"5min":241.61500000000001,"15min":241.61500000000001},"last":2.073}]},{"osd":2,"last update":"Thu Oct 29 09:10:34 2020","interfaces":[{"interface":"back","average":{"1min":2.0310000000000001,"5min":5.5650000000000004,"15min":21.861000000000001},"min":{"1min":1.444,"5min":0.94899999999999995,"15min":0.94899999999999995},"max":{"1min":2.9409999999999998,"5min":208.97399999999999,"15min":503.85000000000002},"last":2.1019999999999999},{"interface":"front","average":{"1min":60.600999999999999,"5min":14.031000000000001,"15min":34.015000000000001},"min":{"1min":1.8109999999999999,"5min":1.169,"15min":1.169},"max":{"1min":540.70500000000004,"5min":540.70500000000004,"15min":540.70500000000004},"last":2.9209999999999998}]},{"osd":3,"last update":"Thu Oct 29 09:10:34 2020","interfaces":[{"interface":"back","average":{"1min":2.1240000000000001,"5min":9.7430000000000003,"15min":4.9809999999999999},"min":{"1min":1.3149999999999999,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":3.8559999999999999,"5min":770.60400000000004,"15min":770.60400000000004},"last":2.23},{"interface":"front","average":{"1min":2.343,"5min":12.279,"15min":5.7510000000000003},"min":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"max":{"1min":3.089,"5min":937.09900000000005,"15min":937.09900000000005},"last":3.048}]},{"osd":4,"last update":"Thu Oct 29 09:10:34 2020","interfaces":[{"interface":"back","average":{"1min":34.405999999999999,"5min":13.635999999999999,"15min":6.3319999999999999},"min":{"1min":1.669,"5min":1.2070000000000001,"15min":1.2070000000000001},"max":{"1min":607.44399999999996,"5min":607.44399999999996,"15min":607.44399999999996},"last":2.036},{"interface":"front","average":{"1min":2.8039999999999998,"5min":9.4280000000000008,"15min":4.9580000000000002},"min":{"1min":2.117,"5min":1.1220000000000001,"15min":1.1220000000000001},"max":{"1min":4.0449999999999999,"5min":606.85299999999995,"15min":606.85299999999995},"last":2.1890000000000001}]},{"osd":6,"last update":"Thu Oct 29 09:07:31 2020","interfaces":[{"interface":"back","average":{"1min":4.4660000000000002,"5min":3.4169999999999998,"15min":3.2850000000000001},"min":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"max":{"1min":28.344000000000001,"5min":28.344000000000001,"15min":28.344000000000001},"last":3.1070000000000002},{"interface":"front","average":{"1min":2.7240000000000002,"5min":5.806,"15min":3.8719999999999999},"min":{"1min":1.6399999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"max":{"1min":3.5409999999999999,"5min":330.59100000000001,"15min":330.59100000000001},"last":2.625}]},{"osd":7,"last update":"Thu Oct 29 09:07:31 2020","interfaces":[{"interface":"back","average":{"1min":2.6110000000000002,"5min":3.2040000000000002,"15min":3.3940000000000001},"min":{"1min":1.4059999999999999,"5min":1.159,"15min":1.159},"max":{"1min":3.7080000000000002,"5min":10.272,"15min":10.272},"last":2.9870000000000001},{"interface":"front","average":{"1min":3.363,"5min":3.2559999999999998,"15min":3.2669999999999999},"min":{"1min":1.6659999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"max":{"1min":4.4969999999999999,"5min":5.7060000000000004,"15min":5.7060000000000004},"last":2.5920000000000001}]}]}]}} , 2020-10-29T09:10:43.080 INFO:tasks.ceph.ceph_manager.ceph:recovered! , 2020-10-29T09:10:43.081 INFO:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early config rm mgr mgr_debug_aggressive_pg_num_changes 2020-10-29T22:44:43.120 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:10:50.210 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2020-10-29T09:10:31.199149+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=9, tm_min=10, tm_sec=31, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=9, tm_min=10, tm_sec=44, tm_wday=3, tm_yday=303, tm_isdst=0) , 2020-10-29T09:10:50.210 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2020-10-29T09:10:13.036105+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=9, tm_min=10, tm_sec=13, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=9, tm_min=10, tm_sec=44, tm_wday=3, tm_yday=303, tm_isdst=0) , 2020-10-29T09:10:50.211 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2020-10-29T09:10:14.907450+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=9, tm_min=10, tm_sec=14, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=9, tm_min=10, tm_sec=44, tm_wday=3, tm_yday=303, tm_isdst=0) 2020-10-29T22:44:43.121 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:11:32.054 INFO:tasks.ceph.mon.c:Stopped , 2020-10-29T09:11:32.054 DEBUG:tasks.ceph.mon.b:waiting for process to exit , 2020-10-29T09:11:32.055 INFO:teuthology.orchestra.run:waiting for 300 2020-10-29T22:44:43.122 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:12:00.229 INFO:teuthology.orchestra.run.smithi002.stdout:-hv02.front.sepi 163.237.218.18 2 u 11 64 377 0.102 -1.780 4.152 , 2020-10-29T09:12:00.229 INFO:teuthology.orchestra.run.smithi002.stdout:*hv03.front.sepi 74.6.168.72 3 u 29 64 377 0.113 -1.949 3.309 , 2020-10-29T09:12:00.230 INFO:teuthology.orchestra.run.smithi002.stdout:+hv04.front.sepi 144.172.126.201 3 u 28 64 377 0.098 -0.914 3.386 2020-10-29T22:44:43.574 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6268-g944bb62c (944bb62c829d723d3f38fa6a8966ac655e3866f2) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f4da7e56136] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x55790f3ddca6] 2020-10-29T22:44:43.575 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:52.543 INFO:teuthology.orchestra.run.smithi194.stdout: }, , 2020-10-29T10:12:52.544 INFO:teuthology.orchestra.run.smithi194.stdout: { , 2020-10-29T10:12:52.544 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2020-10-29T22:44:43.575 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:52.565 INFO:teuthology.orchestra.run.smithi194.stdout: }, , 2020-10-29T10:12:52.565 INFO:teuthology.orchestra.run.smithi194.stdout: { , 2020-10-29T10:12:52.566 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "waiting for rw locks", 2020-10-29T22:44:43.576 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:52.583 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 2.6057000000000001e-05 , 2020-10-29T10:12:52.583 INFO:teuthology.orchestra.run.smithi194.stdout: }, , 2020-10-29T10:12:52.583 INFO:teuthology.orchestra.run.smithi194.stdout: { 2020-10-29T22:44:43.576 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:52.605 INFO:teuthology.orchestra.run.smithi194.stdout: "type_data": { , 2020-10-29T10:12:52.605 INFO:teuthology.orchestra.run.smithi194.stdout: "flag_point": "reached pg", , 2020-10-29T10:12:52.605 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ 2020-10-29T22:44:43.577 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:52.620 INFO:teuthology.orchestra.run.smithi194.stdout: }, , 2020-10-29T10:12:52.621 INFO:teuthology.orchestra.run.smithi194.stdout: { , 2020-10-29T10:12:52.621 INFO:teuthology.orchestra.run.smithi194.stdout: "description": "osd_op(client.4417.0:1952 3.16s0 3:6e26fb7b:::smithi19426864-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e161)", 2020-10-29T22:44:43.577 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:52.637 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "dispatched", , 2020-10-29T10:12:52.637 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2020-10-29T10:11:41.602946+0000", , 2020-10-29T10:12:52.637 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 6.9140000000000002e-06 2020-10-29T22:44:43.578 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:52.653 INFO:teuthology.orchestra.run.smithi194.stdout: { , 2020-10-29T10:12:52.653 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", , 2020-10-29T10:12:52.653 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2020-10-29T10:11:47.308435+0000", 2020-10-29T22:44:43.578 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:52.668 INFO:teuthology.orchestra.run.smithi194.stdout: { , 2020-10-29T10:12:52.668 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "done", , 2020-10-29T10:12:52.668 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2020-10-29T10:11:47.582490+0000", 2020-10-29T22:44:43.579 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:52.684 INFO:teuthology.orchestra.run.smithi194.stdout: }, , 2020-10-29T10:12:52.684 INFO:teuthology.orchestra.run.smithi194.stdout: { , 2020-10-29T10:12:52.684 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "reached_pg", 2020-10-29T22:44:43.579 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:52.698 INFO:teuthology.orchestra.run.smithi194.stdout: } , 2020-10-29T10:12:52.699 INFO:teuthology.orchestra.run.smithi194.stdout: ] , 2020-10-29T10:12:52.699 INFO:teuthology.orchestra.run.smithi194.stdout: } 2020-10-29T22:44:43.580 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:52.714 INFO:teuthology.orchestra.run.smithi194.stdout: }, , 2020-10-29T10:12:52.714 INFO:teuthology.orchestra.run.smithi194.stdout: "events": [ , 2020-10-29T10:12:52.714 INFO:teuthology.orchestra.run.smithi194.stdout: { 2020-10-29T22:44:43.580 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:52.730 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 9.0100000000000001e-06 , 2020-10-29T10:12:52.730 INFO:teuthology.orchestra.run.smithi194.stdout: }, , 2020-10-29T10:12:52.730 INFO:teuthology.orchestra.run.smithi194.stdout: { 2020-10-29T22:44:43.581 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:52.745 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.10926171699999999 , 2020-10-29T10:12:52.745 INFO:teuthology.orchestra.run.smithi194.stdout: } , 2020-10-29T10:12:52.745 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2020-10-29T22:44:43.581 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:53.257 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.2439999999999999e-05 , 2020-10-29T10:12:53.257 INFO:teuthology.orchestra.run.smithi194.stdout: }, , 2020-10-29T10:12:53.257 INFO:teuthology.orchestra.run.smithi194.stdout: { 2020-10-29T22:44:43.582 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:53.274 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 8.7843999999999994e-05 , 2020-10-29T10:12:53.274 INFO:teuthology.orchestra.run.smithi194.stdout: }, , 2020-10-29T10:12:53.274 INFO:teuthology.orchestra.run.smithi194.stdout: { 2020-10-29T22:44:43.582 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:53.289 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2020-10-29T10:09:59.378501+0000", , 2020-10-29T10:12:53.289 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.74708767499999995 , 2020-10-29T10:12:53.289 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2020-10-29T22:44:43.584 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:53.303 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", , 2020-10-29T10:12:53.303 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2020-10-29T10:10:48.544674+0000", , 2020-10-29T10:12:53.303 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0 2020-10-29T22:44:43.584 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:53.317 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", , 2020-10-29T10:12:53.317 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2020-10-29T10:10:48.544837+0000", , 2020-10-29T10:12:53.318 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0078786540000000006 2020-10-29T22:44:43.585 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:53.331 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "sub_op_committed", , 2020-10-29T10:12:53.331 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2020-10-29T10:11:04.297594+0000", , 2020-10-29T10:12:53.331 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 0.0014536110000000001 2020-10-29T22:44:43.585 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:53.345 INFO:teuthology.orchestra.run.smithi194.stdout: } , 2020-10-29T10:12:53.345 INFO:teuthology.orchestra.run.smithi194.stdout: }, , 2020-10-29T10:12:53.345 INFO:teuthology.orchestra.run.smithi194.stdout: { 2020-10-29T22:44:43.586 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:53.359 INFO:teuthology.orchestra.run.smithi194.stdout: { , 2020-10-29T10:12:53.359 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "initiated", , 2020-10-29T10:12:53.360 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2020-10-29T10:11:05.265767+0000", 2020-10-29T22:44:43.586 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:53.373 INFO:teuthology.orchestra.run.smithi194.stdout: { , 2020-10-29T10:12:53.373 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", , 2020-10-29T10:12:53.373 INFO:teuthology.orchestra.run.smithi194.stdout: "time": "2020-10-29T10:11:05.266407+0000", 2020-10-29T22:44:43.587 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:53.387 INFO:teuthology.orchestra.run.smithi194.stdout: "client_info": { , 2020-10-29T10:12:53.387 INFO:teuthology.orchestra.run.smithi194.stdout: "client": "client.4417", , 2020-10-29T10:12:53.388 INFO:teuthology.orchestra.run.smithi194.stdout: "client_addr": "172.21.15.194:0/1989335687", 2020-10-29T22:44:43.587 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:53.401 INFO:teuthology.orchestra.run.smithi194.stdout: }, , 2020-10-29T10:12:53.402 INFO:teuthology.orchestra.run.smithi194.stdout: { , 2020-10-29T10:12:53.402 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "header_read", 2020-10-29T22:44:43.588 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:53.416 INFO:teuthology.orchestra.run.smithi194.stdout: }, , 2020-10-29T10:12:53.416 INFO:teuthology.orchestra.run.smithi194.stdout: { , 2020-10-29T10:12:53.416 INFO:teuthology.orchestra.run.smithi194.stdout: "event": "queued_for_pg", 2020-10-29T22:44:43.589 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:53.430 INFO:teuthology.orchestra.run.smithi194.stdout: "duration": 5.1375999999999997e-05 , 2020-10-29T10:12:53.430 INFO:teuthology.orchestra.run.smithi194.stdout: }, , 2020-10-29T10:12:53.430 INFO:teuthology.orchestra.run.smithi194.stdout: { 2020-10-29T22:44:43.589 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:53.544+0000 7fb9cc5d8700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 22020) UID: 0 , 2020-10-29T10:12:53.572 INFO:teuthology.orchestra.run.smithi075.stdout:{ , 2020-10-29T10:12:53.573 INFO:teuthology.orchestra.run.smithi075.stdout: "ops": [], 2020-10-29T22:44:43.590 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:53.864 INFO:teuthology.orchestra.run.smithi075.stdout: } , 2020-10-29T10:12:53.864 INFO:teuthology.orchestra.run.smithi075.stdout: ] , 2020-10-29T10:12:53.864 INFO:teuthology.orchestra.run.smithi075.stdout: } 2020-10-29T22:44:43.590 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:53.878 INFO:teuthology.orchestra.run.smithi075.stdout: }, , 2020-10-29T10:12:53.878 INFO:teuthology.orchestra.run.smithi075.stdout: "events": [ , 2020-10-29T10:12:53.878 INFO:teuthology.orchestra.run.smithi075.stdout: { 2020-10-29T22:44:43.591 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:53.893 INFO:teuthology.orchestra.run.smithi075.stdout: "duration": 1.7779999999999999e-05 , 2020-10-29T10:12:53.893 INFO:teuthology.orchestra.run.smithi075.stdout: }, , 2020-10-29T10:12:53.893 INFO:teuthology.orchestra.run.smithi075.stdout: { 2020-10-29T22:44:43.591 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:53.907 INFO:teuthology.orchestra.run.smithi075.stdout: { , 2020-10-29T10:12:53.908 INFO:teuthology.orchestra.run.smithi075.stdout: "description": "osd_op(client.4417.0:1580 3.8s0 3.f2eef768 (undecoded) ondisk+write+known_if_redirected e123)", , 2020-10-29T10:12:53.908 INFO:teuthology.orchestra.run.smithi075.stdout: "initiated_at": "2020-10-29T10:11:03.292114+0000", 2020-10-29T22:44:43.592 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:53.922 INFO:teuthology.orchestra.run.smithi075.stdout: "time": "2020-10-29T10:11:03.292728+0000", , 2020-10-29T10:12:53.922 INFO:teuthology.orchestra.run.smithi075.stdout: "duration": 3.1350000000000001e-06 , 2020-10-29T10:12:53.922 INFO:teuthology.orchestra.run.smithi075.stdout: }, 2020-10-29T22:44:43.592 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:53.936 INFO:teuthology.orchestra.run.smithi075.stdout: "duration": 0.98510730099999999, , 2020-10-29T10:12:53.937 INFO:teuthology.orchestra.run.smithi075.stdout: "type_data": { , 2020-10-29T10:12:53.937 INFO:teuthology.orchestra.run.smithi075.stdout: "flag_point": "reached pg", 2020-10-29T22:44:43.593 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:53.951 INFO:teuthology.orchestra.run.smithi075.stdout: "event": "queued_for_pg", , 2020-10-29T10:12:53.951 INFO:teuthology.orchestra.run.smithi075.stdout: "time": "2020-10-29T10:11:03.293094+0000", , 2020-10-29T10:12:53.951 INFO:teuthology.orchestra.run.smithi075.stdout: "duration": 0.00015803599999999999 2020-10-29T22:44:43.593 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:53.966 INFO:teuthology.orchestra.run.smithi075.stdout: "event": "commit_sent", , 2020-10-29T10:12:53.966 INFO:teuthology.orchestra.run.smithi075.stdout: "time": "2020-10-29T10:11:04.297930+0000", , 2020-10-29T10:12:53.966 INFO:teuthology.orchestra.run.smithi075.stdout: "duration": 0.032230648000000001 2020-10-29T22:44:43.596 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:53.980 INFO:teuthology.orchestra.run.smithi075.stdout: "age": 110.526397816, , 2020-10-29T10:12:53.981 INFO:teuthology.orchestra.run.smithi075.stdout: "duration": 1.009644218, , 2020-10-29T10:12:53.981 INFO:teuthology.orchestra.run.smithi075.stdout: "type_data": { 2020-10-29T22:44:43.597 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:53.995 INFO:teuthology.orchestra.run.smithi075.stdout: { , 2020-10-29T10:12:53.995 INFO:teuthology.orchestra.run.smithi075.stdout: "event": "throttled", , 2020-10-29T10:12:53.995 INFO:teuthology.orchestra.run.smithi075.stdout: "time": "2020-10-29T10:11:03.321309+0000", 2020-10-29T22:44:43.598 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:54.009 INFO:teuthology.orchestra.run.smithi075.stdout: { , 2020-10-29T10:12:54.009 INFO:teuthology.orchestra.run.smithi075.stdout: "event": "dispatched", , 2020-10-29T10:12:54.009 INFO:teuthology.orchestra.run.smithi075.stdout: "time": "2020-10-29T10:11:03.321433+0000", 2020-10-29T22:44:43.598 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:54.024 INFO:teuthology.orchestra.run.smithi075.stdout: { , 2020-10-29T10:12:54.024 INFO:teuthology.orchestra.run.smithi075.stdout: "event": "reached_pg", , 2020-10-29T10:12:54.024 INFO:teuthology.orchestra.run.smithi075.stdout: "time": "2020-10-29T10:11:04.286714+0000", 2020-10-29T22:44:43.599 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:54.038 INFO:teuthology.orchestra.run.smithi075.stdout: { , 2020-10-29T10:12:54.038 INFO:teuthology.orchestra.run.smithi075.stdout: "event": "waiting for rw locks", , 2020-10-29T10:12:54.039 INFO:teuthology.orchestra.run.smithi075.stdout: "time": "2020-10-29T10:11:04.329961+0000", 2020-10-29T22:44:43.600 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:54.052 INFO:teuthology.orchestra.run.smithi075.stdout: "client_info": { , 2020-10-29T10:12:54.053 INFO:teuthology.orchestra.run.smithi075.stdout: "client": "client.4417", , 2020-10-29T10:12:54.053 INFO:teuthology.orchestra.run.smithi075.stdout: "client_addr": "172.21.15.194:0/1989335687", 2020-10-29T22:44:43.600 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:12:54.067 INFO:teuthology.orchestra.run.smithi075.stdout: }, , 2020-10-29T10:12:54.067 INFO:teuthology.orchestra.run.smithi075.stdout: { , 2020-10-29T10:12:54.067 INFO:teuthology.orchestra.run.smithi075.stdout: "event": "queued_for_pg", 2020-10-29T22:44:43.605 WARNING:scrape:Ignoring malparsed backtrace: osd.5: osd_enable_op_tracker = 'false' , osd.6: osd_enable_op_tracker = 'false' , osd.7: osd_enable_op_tracker = 'false' 2020-10-29T22:44:43.608 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:13:23.629 INFO:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd primary-affinity 4 1 , set osd.4 primary-affinity to 1 (8655362) , 2020-10-29T10:13:24.639 INFO:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd reweight 5 1 2020-10-29T22:44:43.609 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:13:40.354 INFO:teuthology.orchestra.run.smithi194.stdout:{ , 2020-10-29T10:13:40.355 INFO:teuthology.orchestra.run.smithi194.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2020-10-29T10:13:40.355 INFO:teuthology.orchestra.run.smithi194.stdout:} 2020-10-29T22:44:43.613 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:13:43.838 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2020-10-29T10:12:52.434969+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=10, tm_min=12, tm_sec=52, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=10, tm_min=13, tm_sec=38, tm_wday=3, tm_yday=303, tm_isdst=0) , 2020-10-29T10:13:43.839 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2020-10-29T10:12:29.118851+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=10, tm_min=12, tm_sec=29, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=10, tm_min=13, tm_sec=38, tm_wday=3, tm_yday=303, tm_isdst=0) , 2020-10-29T10:13:43.839 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2020-10-29T10:12:57.204073+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=10, tm_min=12, tm_sec=57, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=10, tm_min=13, tm_sec=38, tm_wday=3, tm_yday=303, tm_isdst=0) 2020-10-29T22:44:43.615 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:14:24.535 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2020-10-29T10:12:28.544937+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=10, tm_min=12, tm_sec=28, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=10, tm_min=13, tm_sec=38, tm_wday=3, tm_yday=303, tm_isdst=0) , 2020-10-29T10:14:24.536 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2020-10-29T10:13:17.924505+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=10, tm_min=13, tm_sec=17, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=10, tm_min=13, tm_sec=38, tm_wday=3, tm_yday=303, tm_isdst=0) , 2020-10-29T10:14:24.536 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2020-10-29T22:44:43.616 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:15:07.266 INFO:teuthology.orchestra.run.smithi075:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 , 2020-10-29T10:15:07.340 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi075.front.sepia.ceph.com , 2020-10-29T10:15:07.341 INFO:teuthology.orchestra.run.smithi075:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2020-10-29T22:44:43.617 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:15:40.350 INFO:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' , 2020-10-29T10:15:40.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 , 2020-10-29T10:15:40.394 INFO:teuthology.orchestra.run.smithi075:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' 2020-10-29T22:44:43.618 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T10:16:06.111 INFO:teuthology.orchestra.run.smithi075:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest , 2020-10-29T10:16:06.126 INFO:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest , 2020-10-29T10:16:06.137 INFO:teuthology.orchestra.run.smithi075.stdout: 262185 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 29 10:15 /home/ubuntu/cephtest 2020-10-29T22:44:43.956 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6268-g944bb62c (944bb62c829d723d3f38fa6a8966ac655e3866f2) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7ff0620de9f6] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x275c10) [0x7ff0620dec10] 2020-10-29T22:44:43.957 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:13.301 INFO:teuthology.orchestra.run.smithi028.stdout: } , 2020-10-29T08:45:13.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, , 2020-10-29T08:45:13.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2020-10-29T22:44:43.957 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:13.325 INFO:teuthology.orchestra.run.smithi028.stdout: { , 2020-10-29T08:45:13.325 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", , 2020-10-29T08:45:13.325 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2020-10-29T08:44:13.518624+0000", 2020-10-29T22:44:43.958 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:13.340 INFO:teuthology.orchestra.run.smithi028.stdout: { , 2020-10-29T08:45:13.340 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", , 2020-10-29T08:45:13.340 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2020-10-29T08:44:13.520815+0000", 2020-10-29T22:44:43.958 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:13.355 INFO:teuthology.orchestra.run.smithi028.stdout: ] , 2020-10-29T08:45:13.355 INFO:teuthology.orchestra.run.smithi028.stdout: } , 2020-10-29T08:45:13.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2020-10-29T22:44:43.959 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:13.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, , 2020-10-29T08:45:13.369 INFO:teuthology.orchestra.run.smithi028.stdout: { , 2020-10-29T08:45:13.369 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2020-10-29T22:44:43.959 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:13.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, , 2020-10-29T08:45:13.383 INFO:teuthology.orchestra.run.smithi028.stdout: { , 2020-10-29T08:45:13.383 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_started", 2020-10-29T22:44:43.960 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:13.397 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", , 2020-10-29T08:45:13.397 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { , 2020-10-29T08:45:13.397 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.4431", 2020-10-29T22:44:43.961 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:13.410 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9179999999999997e-06 , 2020-10-29T08:45:13.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, , 2020-10-29T08:45:13.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2020-10-29T22:44:43.961 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:13.423 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2020-10-29T08:45:08.528535+0000", , 2020-10-29T08:45:13.424 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6215e-05 , 2020-10-29T08:45:13.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2020-10-29T22:44:43.961 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:13.438 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2020-10-29T08:45:08.528751+0000", , 2020-10-29T08:45:13.438 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3439999999999999e-06 , 2020-10-29T08:45:13.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2020-10-29T22:44:43.962 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:13.452 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2020-10-29T08:45:09.539250+0000", , 2020-10-29T08:45:13.452 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013233048000000001 , 2020-10-29T08:45:13.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2020-10-29T22:44:43.962 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:13.466 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2020-10-29T08:45:09.649007+0000", , 2020-10-29T08:45:13.466 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00049942300000000003 , 2020-10-29T08:45:13.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2020-10-29T22:44:43.963 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:13.480 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.50933192299999996, , 2020-10-29T08:45:13.480 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { , 2020-10-29T08:45:13.480 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2020-10-29T22:44:43.964 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:13.493 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", , 2020-10-29T08:45:13.493 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2020-10-29T08:45:10.125530+0000", , 2020-10-29T08:45:13.493 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000113872 2020-10-29T22:44:43.964 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:13.508 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for rw locks", , 2020-10-29T08:45:13.508 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2020-10-29T08:45:10.273692+0000", , 2020-10-29T08:45:13.508 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.21047465100000001 2020-10-29T22:44:43.965 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:13.984 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 3.4750000000000002e-06 , 2020-10-29T08:45:13.984 INFO:teuthology.orchestra.run.smithi051.stdout: }, , 2020-10-29T08:45:13.984 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-10-29T22:44:43.965 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:14.003 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 1.3995999999999999e-05 , 2020-10-29T08:45:14.003 INFO:teuthology.orchestra.run.smithi051.stdout: }, , 2020-10-29T08:45:14.003 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-10-29T22:44:43.966 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:14.039 INFO:teuthology.orchestra.run.smithi051.stdout: { , 2020-10-29T08:45:14.039 INFO:teuthology.orchestra.run.smithi051.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 94/22 ECSubWriteReply(tid=147, last_complete=94'55, committed=1, applied=1))", , 2020-10-29T08:45:14.039 INFO:teuthology.orchestra.run.smithi051.stdout: "initiated_at": "2020-10-29T08:45:04.997719+0000", 2020-10-29T22:44:43.966 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:14.054 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:45:08.499718+0000", , 2020-10-29T08:45:14.054 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 0.0060316129999999999 , 2020-10-29T08:45:14.054 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-10-29T22:44:43.967 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:14.069 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:45:08.510520+0000", , 2020-10-29T08:45:14.069 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 0.0059572679999999999 , 2020-10-29T08:45:14.069 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-10-29T22:44:43.967 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:14.083 INFO:teuthology.orchestra.run.smithi051.stdout: }, , 2020-10-29T08:45:14.083 INFO:teuthology.orchestra.run.smithi051.stdout: { , 2020-10-29T08:45:14.083 INFO:teuthology.orchestra.run.smithi051.stdout: "description": "osd_op(client.4431.0:1006 3.9s0 3:9425d2f8:::smithi02826543-49:head [setxattr _header (57) in=64b] snapc 46=[46,45,41,3e,37,33,30,2c,26,1e] ondisk+write+known_if_redirected e98)", 2020-10-29T22:44:43.968 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:14.098 INFO:teuthology.orchestra.run.smithi051.stdout: "event": "header_read", , 2020-10-29T08:45:14.098 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:45:08.545088+0000", , 2020-10-29T08:45:14.098 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 3.2990000000000001e-06 2020-10-29T22:44:43.968 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:14.112 INFO:teuthology.orchestra.run.smithi051.stdout: "event": "reached_pg", , 2020-10-29T08:45:14.112 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:45:09.540948+0000", , 2020-10-29T08:45:14.113 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 0.00026468100000000001 2020-10-29T22:44:43.969 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:14.132 INFO:teuthology.orchestra.run.smithi051.stdout: "age": 5.4256935970000004, , 2020-10-29T08:45:14.132 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 1.00018714, , 2020-10-29T08:45:14.132 INFO:teuthology.orchestra.run.smithi051.stdout: "type_data": { 2020-10-29T22:44:43.969 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:14.146 INFO:teuthology.orchestra.run.smithi051.stdout: { , 2020-10-29T08:45:14.147 INFO:teuthology.orchestra.run.smithi051.stdout: "event": "dispatched", , 2020-10-29T08:45:14.147 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:45:08.795965+0000", 2020-10-29T22:44:43.970 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:14.165 INFO:teuthology.orchestra.run.smithi051.stdout: }, , 2020-10-29T08:45:14.165 INFO:teuthology.orchestra.run.smithi051.stdout: { , 2020-10-29T08:45:14.166 INFO:teuthology.orchestra.run.smithi051.stdout: "event": "throttled", 2020-10-29T22:44:43.972 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:14.184 INFO:teuthology.orchestra.run.smithi051.stdout: }, , 2020-10-29T08:45:14.184 INFO:teuthology.orchestra.run.smithi051.stdout: { , 2020-10-29T08:45:14.184 INFO:teuthology.orchestra.run.smithi051.stdout: "event": "reached_pg", 2020-10-29T22:44:43.972 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:14.198 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 0 , 2020-10-29T08:45:14.198 INFO:teuthology.orchestra.run.smithi051.stdout: }, , 2020-10-29T08:45:14.198 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-10-29T22:44:43.973 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:14.221 INFO:teuthology.orchestra.run.smithi051.stdout: { , 2020-10-29T08:45:14.221 INFO:teuthology.orchestra.run.smithi051.stdout: "description": "MOSDECSubOpRead(3.fs1 102/22 ECSubRead(tid=597, to_read={3:fc834db7:::smithi02826543-25:head=0,4096,0}, subchunks={3:fc834db7:::smithi02826543-25:head=[0,1]}, attrs_to_read=))", , 2020-10-29T08:45:14.221 INFO:teuthology.orchestra.run.smithi051.stdout: "initiated_at": "2020-10-29T08:45:12.540064+0000", 2020-10-29T22:44:43.974 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:14.689 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:44:03.549351+0000", , 2020-10-29T08:45:14.690 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 9.8509999999999994e-06 , 2020-10-29T08:45:14.690 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-10-29T22:44:43.974 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:14.704 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:44:03.553027+0000", , 2020-10-29T08:45:14.704 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 0.010566684 , 2020-10-29T08:45:14.705 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-10-29T22:44:43.975 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:14.720 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 0.19933210700000001, , 2020-10-29T08:45:14.720 INFO:teuthology.orchestra.run.smithi051.stdout: "type_data": { , 2020-10-29T08:45:14.720 INFO:teuthology.orchestra.run.smithi051.stdout: "flag_point": "started", 2020-10-29T22:44:43.975 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:14.735 INFO:teuthology.orchestra.run.smithi051.stdout: "event": "started", , 2020-10-29T08:45:14.735 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:44:19.671166+0000", , 2020-10-29T08:45:14.735 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 0.01228813 2020-10-29T22:44:43.976 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:14.750 INFO:teuthology.orchestra.run.smithi051.stdout: "event": "done", , 2020-10-29T08:45:14.750 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:44:19.689693+0000", , 2020-10-29T08:45:14.750 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 0.018643980000000001 2020-10-29T22:44:43.976 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:14.771 INFO:teuthology.orchestra.run.smithi051.stdout: { , 2020-10-29T08:45:14.771 INFO:teuthology.orchestra.run.smithi051.stdout: "event": "dispatched", , 2020-10-29T08:45:14.772 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:44:23.657732+0000", 2020-10-29T22:44:43.978 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:14.787 INFO:teuthology.orchestra.run.smithi051.stdout: { , 2020-10-29T08:45:14.788 INFO:teuthology.orchestra.run.smithi051.stdout: "event": "commit_sent", , 2020-10-29T08:45:14.788 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:44:33.497228+0000", 2020-10-29T22:44:43.979 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:14.802 INFO:teuthology.orchestra.run.smithi051.stdout: }, , 2020-10-29T08:45:14.803 INFO:teuthology.orchestra.run.smithi051.stdout: { , 2020-10-29T08:45:14.803 INFO:teuthology.orchestra.run.smithi051.stdout: "event": "all_read", 2020-10-29T22:44:43.980 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:14.816 INFO:teuthology.orchestra.run.smithi051.stdout: "flag_point": "reached pg", , 2020-10-29T08:45:14.816 INFO:teuthology.orchestra.run.smithi051.stdout: "events": [ , 2020-10-29T08:45:14.817 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-10-29T22:44:43.980 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:14.830 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 0.10733277300000001 , 2020-10-29T08:45:14.830 INFO:teuthology.orchestra.run.smithi051.stdout: } , 2020-10-29T08:45:14.830 INFO:teuthology.orchestra.run.smithi051.stdout: ] 2020-10-29T22:44:43.981 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:14.844 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:45:03.491934+0000", , 2020-10-29T08:45:14.844 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 1.4698999999999999e-05 , 2020-10-29T08:45:14.844 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-10-29T22:44:43.981 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:14.860 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:45:04.059106+0000", , 2020-10-29T08:45:14.860 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 3.9369000000000002e-05 , 2020-10-29T08:45:14.860 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-10-29T22:44:43.981 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:14.874 INFO:teuthology.orchestra.run.smithi051.stdout: "event": "all_read", , 2020-10-29T08:45:14.874 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:45:04.523544+0000", , 2020-10-29T08:45:14.874 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 1.4039999999999999e-05 2020-10-29T22:44:43.982 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:15.173 INFO:teuthology.orchestra.run.smithi051.stdout:{ , 2020-10-29T08:45:15.173 INFO:teuthology.orchestra.run.smithi051.stdout: "ops": [], , 2020-10-29T08:45:15.174 INFO:teuthology.orchestra.run.smithi051.stdout: "complaint_time": 30, 2020-10-29T22:44:43.982 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:15.392 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:44:23.651516+0000", , 2020-10-29T08:45:15.393 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 0 , 2020-10-29T08:45:15.394 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-10-29T22:44:43.983 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:15.426 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:44:23.656507+0000", , 2020-10-29T08:45:15.429 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 0.000305545 , 2020-10-29T08:45:15.430 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-10-29T22:44:43.983 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:15.450 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:44:23.669317+0000", , 2020-10-29T08:45:15.450 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 0.00018745099999999999 , 2020-10-29T08:45:15.450 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-10-29T22:44:43.984 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:15.464 INFO:teuthology.orchestra.run.smithi051.stdout: "event": "initiated", , 2020-10-29T08:45:15.464 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:44:23.656839+0000", , 2020-10-29T08:45:15.464 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 0 2020-10-29T22:44:43.984 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:15.485 INFO:teuthology.orchestra.run.smithi051.stdout: "event": "reached_pg", , 2020-10-29T08:45:15.485 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:44:43.509131+0000", , 2020-10-29T08:45:15.485 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 0.00010503800000000001 2020-10-29T22:44:43.985 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:15.503 INFO:teuthology.orchestra.run.smithi051.stdout: { , 2020-10-29T08:45:15.503 INFO:teuthology.orchestra.run.smithi051.stdout: "event": "header_read", , 2020-10-29T08:45:15.503 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:44:43.508975+0000", 2020-10-29T22:44:43.985 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:15.518 INFO:teuthology.orchestra.run.smithi051.stdout: }, , 2020-10-29T08:45:15.518 INFO:teuthology.orchestra.run.smithi051.stdout: { , 2020-10-29T08:45:15.519 INFO:teuthology.orchestra.run.smithi051.stdout: "event": "throttled", 2020-10-29T22:44:43.989 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:15.536 INFO:teuthology.orchestra.run.smithi051.stdout: }, , 2020-10-29T08:45:15.536 INFO:teuthology.orchestra.run.smithi051.stdout: { , 2020-10-29T08:45:15.536 INFO:teuthology.orchestra.run.smithi051.stdout: "event": "commit_sent", 2020-10-29T22:44:43.989 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:15.561 INFO:teuthology.orchestra.run.smithi051.stdout: }, , 2020-10-29T08:45:15.561 INFO:teuthology.orchestra.run.smithi051.stdout: "events": [ , 2020-10-29T08:45:15.561 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-10-29T22:44:43.990 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:15.582 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 5.8490000000000002e-06 , 2020-10-29T08:45:15.583 INFO:teuthology.orchestra.run.smithi051.stdout: }, , 2020-10-29T08:45:15.583 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-10-29T22:44:43.991 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:15.610 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 0.001584542 , 2020-10-29T08:45:15.610 INFO:teuthology.orchestra.run.smithi051.stdout: }, , 2020-10-29T08:45:15.610 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-10-29T22:44:43.992 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:15.626 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:44:49.567774+0000", , 2020-10-29T08:45:15.626 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 0 , 2020-10-29T08:45:15.626 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-10-29T22:44:43.992 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:15.641 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:44:49.578353+0000", , 2020-10-29T08:45:15.641 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 0.0054109129999999998 , 2020-10-29T08:45:15.641 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-10-29T22:44:43.993 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:15.939 INFO:teuthology.orchestra.run.smithi051.stdout:{ , 2020-10-29T08:45:15.940 INFO:teuthology.orchestra.run.smithi051.stdout: "ops": [], , 2020-10-29T08:45:15.940 INFO:teuthology.orchestra.run.smithi051.stdout: "complaint_time": 30, 2020-10-29T22:44:43.993 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:16.103 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:43:59.538047+0000", , 2020-10-29T08:45:16.104 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 7.5360000000000004e-06 , 2020-10-29T08:45:16.104 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-10-29T22:44:43.994 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:16.122 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:43:59.572921+0000", , 2020-10-29T08:45:16.122 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 0.0029567930000000001 , 2020-10-29T08:45:16.122 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-10-29T22:44:43.995 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:16.142 INFO:teuthology.orchestra.run.smithi051.stdout: "event": "initiated", , 2020-10-29T08:45:16.142 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:43:59.574085+0000", , 2020-10-29T08:45:16.142 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 0 2020-10-29T22:44:43.995 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:16.162 INFO:teuthology.orchestra.run.smithi051.stdout: "event": "started", , 2020-10-29T08:45:16.162 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:44:14.484984+0000", , 2020-10-29T08:45:16.163 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 0.0071520580000000002 2020-10-29T22:44:43.996 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:16.177 INFO:teuthology.orchestra.run.smithi051.stdout: "event": "reached_pg", , 2020-10-29T08:45:16.177 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:44:14.513235+0000", , 2020-10-29T08:45:16.177 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 9.993e-05 2020-10-29T22:44:43.996 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:16.192 INFO:teuthology.orchestra.run.smithi051.stdout: "client": "client.4431", , 2020-10-29T08:45:16.192 INFO:teuthology.orchestra.run.smithi051.stdout: "client_addr": "172.21.15.28:0/2773662444", , 2020-10-29T08:45:16.193 INFO:teuthology.orchestra.run.smithi051.stdout: "tid": 402 2020-10-29T22:44:44.003 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:16.208 INFO:teuthology.orchestra.run.smithi051.stdout: { , 2020-10-29T08:45:16.208 INFO:teuthology.orchestra.run.smithi051.stdout: "event": "dispatched", , 2020-10-29T08:45:16.208 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:44:14.494773+0000", 2020-10-29T22:44:44.010 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:16.222 INFO:teuthology.orchestra.run.smithi051.stdout: "initiated_at": "2020-10-29T08:44:43.508917+0000", , 2020-10-29T08:45:16.222 INFO:teuthology.orchestra.run.smithi051.stdout: "age": 32.573615525000001, , 2020-10-29T08:45:16.222 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 11.147158642999999, 2020-10-29T22:44:44.010 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:16.235 INFO:teuthology.orchestra.run.smithi051.stdout: }, , 2020-10-29T08:45:16.236 INFO:teuthology.orchestra.run.smithi051.stdout: { , 2020-10-29T08:45:16.236 INFO:teuthology.orchestra.run.smithi051.stdout: "event": "dispatched", 2020-10-29T22:44:44.011 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:16.249 INFO:teuthology.orchestra.run.smithi051.stdout: }, , 2020-10-29T08:45:16.249 INFO:teuthology.orchestra.run.smithi051.stdout: { , 2020-10-29T08:45:16.250 INFO:teuthology.orchestra.run.smithi051.stdout: "event": "started", 2020-10-29T22:44:44.011 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:16.264 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 5.6966000000000003e-05 , 2020-10-29T08:45:16.264 INFO:teuthology.orchestra.run.smithi051.stdout: }, , 2020-10-29T08:45:16.264 INFO:teuthology.orchestra.run.smithi051.stdout: { 2020-10-29T22:44:44.012 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:16.278 INFO:teuthology.orchestra.run.smithi051.stdout: "type_data": { , 2020-10-29T08:45:16.278 INFO:teuthology.orchestra.run.smithi051.stdout: "flag_point": "started", , 2020-10-29T08:45:16.278 INFO:teuthology.orchestra.run.smithi051.stdout: "client_info": { 2020-10-29T22:44:44.013 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:16.292 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:44:59.599298+0000", , 2020-10-29T08:45:16.292 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 2.5660000000000001e-06 , 2020-10-29T08:45:16.292 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-10-29T22:44:44.013 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:16.305 INFO:teuthology.orchestra.run.smithi051.stdout: "time": "2020-10-29T08:45:09.529557+0000", , 2020-10-29T08:45:16.306 INFO:teuthology.orchestra.run.smithi051.stdout: "duration": 0.00032205899999999997 , 2020-10-29T08:45:16.306 INFO:teuthology.orchestra.run.smithi051.stdout: }, 2020-10-29T22:44:44.015 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:37.297 INFO:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.5 flush_pg_stats , 2020-10-29T08:45:37.468 INFO:teuthology.orchestra.run.smithi028.stdout:60129542172 , 2020-10-29T08:45:37.468 INFO:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.6 flush_pg_stats 2020-10-29T22:44:44.017 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:45:48.081 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2020-10-29T08:45:48.082 INFO:teuthology.orchestra.run.smithi028.stdout:} , 2020-10-29T08:45:48.092 INFO:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd deep-scrub 6 2020-10-29T22:44:44.018 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T08:46:30.562 INFO:tasks.ceph.osd.3:Stopped , 2020-10-29T08:46:30.563 DEBUG:tasks.ceph.osd.4:waiting for process to exit , 2020-10-29T08:46:30.563 INFO:teuthology.orchestra.run:waiting for 300 2020-10-29T22:44:44.019 WARNING:scrape:Ignoring malparsed backtrace: next(self.gen) , File "/home/teuthworker/src/github.com_bigwhite37_ceph_pg-ec-primary-write/qa/tasks/rados.py", line 277, in task , running.get() 2020-10-29T22:44:44.021 WARNING:scrape:Ignoring malparsed backtrace: File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ , next(self.gen) , File "/home/teuthworker/src/github.com_bigwhite37_ceph_pg-ec-primary-write/qa/tasks/rados.py", line 277, in task 2020-10-29T22:44:45.010 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-6268-g944bb62c (944bb62c829d723d3f38fa6a8966ac655e3866f2) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f4e4ef50cfb] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x5597c6cf69b0] 2020-10-29T22:44:45.011 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:42.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, , 2020-10-29T09:34:42.963 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:42.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2020-10-29T22:44:45.012 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:42.982 INFO:teuthology.orchestra.run.smithi173.stdout: }, , 2020-10-29T09:34:42.982 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:42.982 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2020-10-29T22:44:45.012 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:43.360 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", , 2020-10-29T09:34:43.361 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:26:17.264142+0000", , 2020-10-29T09:34:43.361 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0068643849999999998 2020-10-29T22:44:45.013 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:43.384 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4409", , 2020-10-29T09:34:43.384 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1665204245", , 2020-10-29T09:34:43.384 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 961 2020-10-29T22:44:45.013 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:43.399 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:43.399 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", , 2020-10-29T09:34:43.400 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:26:07.353723+0000", 2020-10-29T22:44:45.014 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:43.429 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:43.430 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", , 2020-10-29T09:34:43.430 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:30:33.379829+0000", 2020-10-29T22:44:45.014 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:43.449 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:43.449 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", , 2020-10-29T09:34:43.449 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:30:33.385236+0000", 2020-10-29T22:44:45.015 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:43.467 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:43.467 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", , 2020-10-29T09:34:43.468 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:31:18.415940+0000", 2020-10-29T22:44:45.015 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:43.494 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:43.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting_for_map not empty", , 2020-10-29T09:34:43.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:31:18.403230+0000", 2020-10-29T22:44:45.016 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:43.515 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:43.515 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", , 2020-10-29T09:34:43.516 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:31:18.403462+0000", 2020-10-29T22:44:45.016 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:43.541 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:43.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", , 2020-10-29T09:34:43.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:31:18.403795+0000", 2020-10-29T22:44:45.017 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:43.561 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:43.561 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", , 2020-10-29T09:34:43.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:31:18.403998+0000", 2020-10-29T22:44:45.017 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:43.580 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:43.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", , 2020-10-29T09:34:43.580 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:31:37.449830+0000", 2020-10-29T22:44:45.018 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:43.599 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:43.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", , 2020-10-29T09:34:43.600 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:33:33.680705+0000", 2020-10-29T22:44:45.018 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:43.620 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:43.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", , 2020-10-29T09:34:43.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:33:34.517206+0000", 2020-10-29T22:44:45.019 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:43.643 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:43.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", , 2020-10-29T09:34:43.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:33:34.517748+0000", 2020-10-29T22:44:45.019 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:43.663 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:43.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", , 2020-10-29T09:34:43.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:33:35.649450+0000", 2020-10-29T22:44:45.020 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:43.677 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ , 2020-10-29T09:34:43.677 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:43.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2020-10-29T22:44:45.021 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:43.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, , 2020-10-29T09:34:43.691 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:43.691 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2020-10-29T22:44:45.022 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:43.705 INFO:teuthology.orchestra.run.smithi173.stdout: }, , 2020-10-29T09:34:43.705 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:43.705 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2020-10-29T22:44:45.023 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:44.131 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ , 2020-10-29T09:34:44.131 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:44.131 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4409.0:3402 3.4s0 3:27cd6668:::smithi17317291-18:head [copy-from ver 371] snapc 10b=[10b,100,fb,f8,f7,e2,ba,b9,9f] ondisk+write+known_if_redirected e315)", 2020-10-29T22:44:45.023 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:44.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", , 2020-10-29T09:34:44.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:29:54.370670+0000", , 2020-10-29T09:34:44.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2020-10-29T22:44:45.024 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:44.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", , 2020-10-29T09:34:44.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:31:27.542305+0000", , 2020-10-29T09:34:44.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0115999999999999e-05 2020-10-29T22:44:45.025 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:44.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for blocked object", , 2020-10-29T09:34:44.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:31:27.542924+0000", , 2020-10-29T09:34:44.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.050520672000000003 2020-10-29T22:44:45.026 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:44.203 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:44.204 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", , 2020-10-29T09:34:44.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:31:37.459207+0000", 2020-10-29T22:44:45.027 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:44.219 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:44.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", , 2020-10-29T09:34:44.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:31:37.515440+0000", 2020-10-29T22:44:45.027 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:44.233 INFO:teuthology.orchestra.run.smithi173.stdout: }, , 2020-10-29T09:34:44.233 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:44.233 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2020-10-29T22:44:45.028 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:44.248 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", , 2020-10-29T09:34:44.248 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ , 2020-10-29T09:34:44.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2020-10-29T22:44:45.029 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:44.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.065876525000000005 , 2020-10-29T09:34:44.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, , 2020-10-29T09:34:44.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2020-10-29T22:44:45.029 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:44.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:33:39.122655+0000", , 2020-10-29T09:34:44.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4202999999999999e-05 , 2020-10-29T09:34:44.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2020-10-29T22:44:45.030 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:44.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:33:39.123631+0000", , 2020-10-29T09:34:44.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6486340159999999 , 2020-10-29T09:34:44.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2020-10-29T22:44:45.030 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:44.303 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:33:54.549367+0000", , 2020-10-29T09:34:44.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.026987249000000001 , 2020-10-29T09:34:44.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2020-10-29T22:44:45.031 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:44.317 INFO:teuthology.orchestra.run.smithi173.stdout: }, , 2020-10-29T09:34:44.317 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:44.318 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4409.0:5867 3.10s0 3:0c68688d:::smithi17317291-49:head [append 0~0] snapc 1e5=[1e5,1e4,1e0,1dc,1d8,1d7,1d6,1d0,1ce,1cd,1cc,1c7,1a3,134] ondisk+write+known_if_redirected e553)", 2020-10-29T22:44:45.031 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:44.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", , 2020-10-29T09:34:44.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:33:59.413130+0000", , 2020-10-29T09:34:44.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3145000000000005e-05 2020-10-29T22:44:45.032 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:44.479 INFO:teuthology.orchestra.run.smithi173.stdout:{ , 2020-10-29T09:34:44.480 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], , 2020-10-29T09:34:44.480 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2020-10-29T22:44:45.032 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:44.838 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:44.838 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", , 2020-10-29T09:34:44.838 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:30:12.430161+0000", 2020-10-29T22:44:45.033 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:44.853 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { , 2020-10-29T09:34:44.853 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4409", , 2020-10-29T09:34:44.853 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1665204245", 2020-10-29T22:44:45.033 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:44.867 INFO:teuthology.orchestra.run.smithi173.stdout: }, , 2020-10-29T09:34:44.867 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:44.868 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2020-10-29T22:44:45.033 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:44.882 INFO:teuthology.orchestra.run.smithi173.stdout: }, , 2020-10-29T09:34:44.882 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:44.882 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2020-10-29T22:44:45.034 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:44.896 INFO:teuthology.orchestra.run.smithi173.stdout: }, , 2020-10-29T09:34:44.897 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:44.897 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2020-10-29T22:44:45.034 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:44.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1739999999999999e-05 , 2020-10-29T09:34:44.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, , 2020-10-29T09:34:44.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2020-10-29T22:44:45.035 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:44.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00062199900000000003 , 2020-10-29T09:34:44.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, , 2020-10-29T09:34:44.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2020-10-29T22:44:45.035 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:44.945 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { , 2020-10-29T09:34:44.945 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", , 2020-10-29T09:34:44.945 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2020-10-29T22:44:45.036 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:44.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:32:50.465728+0000", , 2020-10-29T09:34:44.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.045519414000000001 , 2020-10-29T09:34:44.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2020-10-29T22:44:45.036 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:44.974 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", , 2020-10-29T09:34:44.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:32:50.467012+0000", , 2020-10-29T09:34:44.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6722e-05 2020-10-29T22:44:45.037 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:44.989 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 84.295523263000007, , 2020-10-29T09:34:44.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.055262292999999997, , 2020-10-29T09:34:44.989 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2020-10-29T22:44:45.037 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:45.003 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2020-10-29T09:33:20.524247+0000", , 2020-10-29T09:34:45.003 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 84.295298775000006, , 2020-10-29T09:34:45.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.055628189000000001, 2020-10-29T22:44:45.037 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:45.019 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.1s1 529/486 ECSubWrite(tid=3452, reqid=osd.6.0:3452, at_version=529'1642, trim_to=0'0, roll_forward_to=527'1640))", , 2020-10-29T09:34:45.019 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2020-10-29T09:33:29.656542+0000", , 2020-10-29T09:34:45.019 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 75.163003986000007, 2020-10-29T22:44:45.038 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:45.407 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:45.408 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", , 2020-10-29T09:34:45.408 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:25:24.235866+0000", 2020-10-29T22:44:45.038 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:45.425 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:45.425 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", , 2020-10-29T09:34:45.425 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:25:24.242217+0000", 2020-10-29T22:44:45.039 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:45.441 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2020-10-29T09:30:08.372163+0000", , 2020-10-29T09:34:45.441 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 277.02408424700002, , 2020-10-29T09:34:45.441 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.029201603, 2020-10-29T22:44:45.039 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:45.457 INFO:teuthology.orchestra.run.smithi173.stdout: }, , 2020-10-29T09:34:45.457 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:45.457 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2020-10-29T22:44:45.040 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:45.477 INFO:teuthology.orchestra.run.smithi173.stdout: }, , 2020-10-29T09:34:45.477 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:45.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2020-10-29T22:44:45.040 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:45.491 INFO:teuthology.orchestra.run.smithi173.stdout: }, , 2020-10-29T09:34:45.491 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:45.491 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2020-10-29T22:44:45.041 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:45.513 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4409.0:5669 3.13s0 3.8b31c653 (undecoded) ondisk+write+known_if_redirected e534)", , 2020-10-29T09:34:45.513 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2020-10-29T09:33:34.537219+0000", , 2020-10-29T09:34:45.513 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 70.859027753000007, 2020-10-29T22:44:45.041 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:45.528 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2830000000000002e-06 , 2020-10-29T09:34:45.528 INFO:teuthology.orchestra.run.smithi173.stdout: }, , 2020-10-29T09:34:45.528 INFO:teuthology.orchestra.run.smithi173.stdout: { 2020-10-29T22:44:45.041 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:45.543 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { , 2020-10-29T09:34:45.543 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", , 2020-10-29T09:34:45.543 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2020-10-29T22:44:45.042 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:45.558 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:33:34.576751+0000", , 2020-10-29T09:34:45.558 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9730000000000002e-05 , 2020-10-29T09:34:45.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2020-10-29T22:44:45.042 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:45.572 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:33:34.576958+0000", , 2020-10-29T09:34:45.572 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.206e-06 , 2020-10-29T09:34:45.573 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2020-10-29T22:44:45.043 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:45.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:33:35.615549+0000", , 2020-10-29T09:34:45.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00033591900000000001 , 2020-10-29T09:34:45.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2020-10-29T22:44:45.043 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:45.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:33:35.644170+0000", , 2020-10-29T09:34:45.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010091805000000001 , 2020-10-29T09:34:45.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2020-10-29T22:44:45.044 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:45.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:33:40.782685+0000", , 2020-10-29T09:34:45.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000126709 , 2020-10-29T09:34:45.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2020-10-29T22:44:45.044 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:45.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2020-10-29T09:33:40.782920+0000", , 2020-10-29T09:34:45.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0031139610000000002 , 2020-10-29T09:34:45.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2020-10-29T22:44:45.045 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:45.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, , 2020-10-29T09:34:45.646 INFO:teuthology.orchestra.run.smithi173.stdout: { , 2020-10-29T09:34:45.646 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4409.0:5817 3.11s0 3:8fa9cc5d:::smithi17317291-15:head [delete] snapc 1e0=[1e0,1dc,1d8,1d7,1d6,1d0,1ce,1cd,1cc,1ca,1c7,1a3,180,134] ondisk+write+peerstat_old+known_if_redirected e548)", 2020-10-29T22:44:45.045 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:45.910 INFO:teuthology.orchestra.run.smithi115.stdout: "complaint_time": 30, , 2020-10-29T09:34:45.910 INFO:teuthology.orchestra.run.smithi115.stdout: "num_blocked_ops": 0 , 2020-10-29T09:34:45.910 INFO:teuthology.orchestra.run.smithi115.stdout:} 2020-10-29T22:44:45.046 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.059 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ , 2020-10-29T09:34:46.059 INFO:teuthology.orchestra.run.smithi115.stdout: { , 2020-10-29T09:34:46.059 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2020-10-29T22:44:45.046 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.077 INFO:teuthology.orchestra.run.smithi115.stdout: }, , 2020-10-29T09:34:46.077 INFO:teuthology.orchestra.run.smithi115.stdout: { , 2020-10-29T09:34:46.077 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2020-10-29T22:44:45.046 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.094 INFO:teuthology.orchestra.run.smithi115.stdout: }, , 2020-10-29T09:34:46.094 INFO:teuthology.orchestra.run.smithi115.stdout: { , 2020-10-29T09:34:46.094 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2020-10-29T22:44:45.047 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.111 INFO:teuthology.orchestra.run.smithi115.stdout: }, , 2020-10-29T09:34:46.111 INFO:teuthology.orchestra.run.smithi115.stdout: { , 2020-10-29T09:34:46.112 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_committed", 2020-10-29T22:44:45.047 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.126 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 , 2020-10-29T09:34:46.126 INFO:teuthology.orchestra.run.smithi115.stdout: }, , 2020-10-29T09:34:46.126 INFO:teuthology.orchestra.run.smithi115.stdout: { 2020-10-29T22:44:45.048 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.141 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5697000000000001e-05 , 2020-10-29T09:34:46.141 INFO:teuthology.orchestra.run.smithi115.stdout: }, , 2020-10-29T09:34:46.141 INFO:teuthology.orchestra.run.smithi115.stdout: { 2020-10-29T22:44:45.048 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.159 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0570000000000001e-06 , 2020-10-29T09:34:46.159 INFO:teuthology.orchestra.run.smithi115.stdout: }, , 2020-10-29T09:34:46.159 INFO:teuthology.orchestra.run.smithi115.stdout: { 2020-10-29T22:44:45.049 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.173 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0009889670000000001 , 2020-10-29T09:34:46.173 INFO:teuthology.orchestra.run.smithi115.stdout: }, , 2020-10-29T09:34:46.173 INFO:teuthology.orchestra.run.smithi115.stdout: { 2020-10-29T22:44:45.049 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.188 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001413627 , 2020-10-29T09:34:46.188 INFO:teuthology.orchestra.run.smithi115.stdout: }, , 2020-10-29T09:34:46.188 INFO:teuthology.orchestra.run.smithi115.stdout: { 2020-10-29T22:44:45.050 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.202 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 4936 , 2020-10-29T09:34:46.202 INFO:teuthology.orchestra.run.smithi115.stdout: }, , 2020-10-29T09:34:46.203 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2020-10-29T22:44:45.050 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.217 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2020-10-29T09:33:39.079416+0000", , 2020-10-29T09:34:46.217 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1446000000000002e-05 , 2020-10-29T09:34:46.217 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2020-10-29T22:44:45.050 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.232 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2020-10-29T09:33:39.079874+0000", , 2020-10-29T09:34:46.232 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4756000000000001e-05 , 2020-10-29T09:34:46.232 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2020-10-29T22:44:45.051 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.261 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2020-10-29T09:33:39.080213+0000", , 2020-10-29T09:34:46.261 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4696999999999994e-05 , 2020-10-29T09:34:46.261 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2020-10-29T22:44:45.051 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.278 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2020-10-29T09:33:39.124163+0000", , 2020-10-29T09:34:46.278 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.38729819599999998 , 2020-10-29T09:34:46.278 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2020-10-29T22:44:45.052 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.295 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2020-10-29T09:33:39.511496+0000", , 2020-10-29T09:34:46.295 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.077e-06 , 2020-10-29T09:34:46.295 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2020-10-29T22:44:45.052 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.309 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2020-10-29T09:33:40.703824+0000", , 2020-10-29T09:34:46.309 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00034603599999999998 , 2020-10-29T09:34:46.309 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2020-10-29T22:44:45.053 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.324 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2020-10-29T09:33:40.774028+0000", , 2020-10-29T09:34:46.324 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6494579060000001 , 2020-10-29T09:34:46.324 INFO:teuthology.orchestra.run.smithi115.stdout: } 2020-10-29T22:44:45.053 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.731 INFO:teuthology.orchestra.run.smithi115.stdout: { , 2020-10-29T09:34:46.731 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", , 2020-10-29T09:34:46.731 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2020-10-29T09:25:40.292304+0000", 2020-10-29T22:44:45.054 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.755 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2020-10-29T09:25:39.296899+0000", , 2020-10-29T09:34:46.755 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 547.41479259699997, , 2020-10-29T09:34:46.755 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0082657260000001, 2020-10-29T22:44:45.054 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.775 INFO:teuthology.orchestra.run.smithi115.stdout: }, , 2020-10-29T09:34:46.775 INFO:teuthology.orchestra.run.smithi115.stdout: { , 2020-10-29T09:34:46.775 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2020-10-29T22:44:45.054 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.791 INFO:teuthology.orchestra.run.smithi115.stdout: }, , 2020-10-29T09:34:46.791 INFO:teuthology.orchestra.run.smithi115.stdout: { , 2020-10-29T09:34:46.791 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2020-10-29T22:44:45.055 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.809 INFO:teuthology.orchestra.run.smithi115.stdout: }, , 2020-10-29T09:34:46.809 INFO:teuthology.orchestra.run.smithi115.stdout: { , 2020-10-29T09:34:46.809 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2020-10-29T22:44:45.055 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.823 INFO:teuthology.orchestra.run.smithi115.stdout: }, , 2020-10-29T09:34:46.824 INFO:teuthology.orchestra.run.smithi115.stdout: { , 2020-10-29T09:34:46.824 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_committed", 2020-10-29T22:44:45.056 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.840 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", , 2020-10-29T09:34:46.840 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { , 2020-10-29T09:34:46.840 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.4409", 2020-10-29T22:44:45.056 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.865 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0620000000000001e-06 , 2020-10-29T09:34:46.865 INFO:teuthology.orchestra.run.smithi115.stdout: }, , 2020-10-29T09:34:46.865 INFO:teuthology.orchestra.run.smithi115.stdout: { 2020-10-29T22:44:45.057 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.882 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8272999999999999e-05 , 2020-10-29T09:34:46.882 INFO:teuthology.orchestra.run.smithi115.stdout: }, , 2020-10-29T09:34:46.882 INFO:teuthology.orchestra.run.smithi115.stdout: { 2020-10-29T22:44:45.057 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.901 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00043919899999999998 , 2020-10-29T09:34:46.901 INFO:teuthology.orchestra.run.smithi115.stdout: }, , 2020-10-29T09:34:46.901 INFO:teuthology.orchestra.run.smithi115.stdout: { 2020-10-29T22:44:45.058 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.916 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001437965 , 2020-10-29T09:34:46.916 INFO:teuthology.orchestra.run.smithi115.stdout: }, , 2020-10-29T09:34:46.916 INFO:teuthology.orchestra.run.smithi115.stdout: { 2020-10-29T22:44:45.058 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.930 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0034893099999999998 , 2020-10-29T09:34:46.930 INFO:teuthology.orchestra.run.smithi115.stdout: }, , 2020-10-29T09:34:46.930 INFO:teuthology.orchestra.run.smithi115.stdout: { 2020-10-29T22:44:45.058 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.944 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.003904018 , 2020-10-29T09:34:46.944 INFO:teuthology.orchestra.run.smithi115.stdout: }, , 2020-10-29T09:34:46.944 INFO:teuthology.orchestra.run.smithi115.stdout: { 2020-10-29T22:44:45.059 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.958 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { , 2020-10-29T09:34:46.958 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", , 2020-10-29T09:34:46.958 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2020-10-29T22:44:45.059 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.972 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2020-10-29T09:33:49.647794+0000", , 2020-10-29T09:34:46.972 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1069999999999998e-05 , 2020-10-29T09:34:46.972 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2020-10-29T22:44:45.060 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.986 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2020-10-29T09:33:53.811345+0000", , 2020-10-29T09:34:46.986 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0375000000000002e-05 , 2020-10-29T09:34:46.986 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2020-10-29T22:44:45.061 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:46.999 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2020-10-29T09:33:54.548427+0000", , 2020-10-29T09:34:47.000 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00033167199999999998 , 2020-10-29T09:34:47.000 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2020-10-29T22:44:45.062 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:47.419 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", , 2020-10-29T09:34:47.419 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2020-10-29T09:28:36.353401+0000", , 2020-10-29T09:34:47.419 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.027974453 2020-10-29T22:44:45.062 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:47.435 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", , 2020-10-29T09:34:47.436 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2020-10-29T09:28:36.399766+0000", , 2020-10-29T09:34:47.436 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.054216251 2020-10-29T22:44:45.063 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:47.450 INFO:teuthology.orchestra.run.smithi115.stdout: { , 2020-10-29T09:34:47.450 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", , 2020-10-29T09:34:47.450 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2020-10-29T09:31:37.463898+0000", 2020-10-29T22:44:45.063 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:47.464 INFO:teuthology.orchestra.run.smithi115.stdout: { , 2020-10-29T09:34:47.464 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_started", , 2020-10-29T09:34:47.464 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2020-10-29T09:32:31.548071+0000", 2020-10-29T22:44:45.064 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:47.478 INFO:teuthology.orchestra.run.smithi115.stdout: }, , 2020-10-29T09:34:47.478 INFO:teuthology.orchestra.run.smithi115.stdout: { , 2020-10-29T09:34:47.478 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2020-10-29T22:44:45.064 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:47.492 INFO:teuthology.orchestra.run.smithi115.stdout: } , 2020-10-29T09:34:47.492 INFO:teuthology.orchestra.run.smithi115.stdout: ] , 2020-10-29T09:34:47.492 INFO:teuthology.orchestra.run.smithi115.stdout: } 2020-10-29T22:44:45.065 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:47.506 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0073360129999999997 , 2020-10-29T09:34:47.506 INFO:teuthology.orchestra.run.smithi115.stdout: }, , 2020-10-29T09:34:47.506 INFO:teuthology.orchestra.run.smithi115.stdout: { 2020-10-29T22:44:45.065 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:47.522 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2020-10-29T09:34:00.536757+0000", , 2020-10-29T09:34:47.522 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0936000000000001e-05 , 2020-10-29T09:34:47.522 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2020-10-29T22:44:45.066 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:47.537 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2020-10-29T09:34:00.604318+0000", , 2020-10-29T09:34:47.537 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.067399905999999996 , 2020-10-29T09:34:47.537 INFO:teuthology.orchestra.run.smithi115.stdout: } 2020-10-29T22:44:45.066 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:47.551 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", , 2020-10-29T09:34:47.551 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2020-10-29T09:34:00.548106+0000", , 2020-10-29T09:34:47.552 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.019696394999999998 2020-10-29T22:44:45.067 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:47.566 INFO:teuthology.orchestra.run.smithi115.stdout: { , 2020-10-29T09:34:47.566 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", , 2020-10-29T09:34:47.566 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2020-10-29T09:34:00.570677+0000", 2020-10-29T22:44:45.067 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:47.581 INFO:teuthology.orchestra.run.smithi115.stdout: { , 2020-10-29T09:34:47.581 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", , 2020-10-29T09:34:47.582 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2020-10-29T09:34:10.459482+0000", 2020-10-29T22:44:45.067 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:47.595 INFO:teuthology.orchestra.run.smithi115.stdout: }, , 2020-10-29T09:34:47.596 INFO:teuthology.orchestra.run.smithi115.stdout: { , 2020-10-29T09:34:47.596 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2020-10-29T22:44:45.068 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:48.000 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2020-10-29T09:34:43.561447+0000", , 2020-10-29T09:34:48.001 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7924999999999999e-05 , 2020-10-29T09:34:48.001 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2020-10-29T22:44:45.068 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:48.015 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", , 2020-10-29T09:34:48.015 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2020-10-29T09:34:43.554264+0000", , 2020-10-29T09:34:48.015 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8900000000000007e-06 2020-10-29T22:44:45.069 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:48.030 INFO:teuthology.orchestra.run.smithi115.stdout: { , 2020-10-29T09:34:48.030 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", , 2020-10-29T09:34:48.030 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2020-10-29T09:34:43.587874+0000", 2020-10-29T22:44:45.069 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:48.043 INFO:teuthology.orchestra.run.smithi115.stdout: { , 2020-10-29T09:34:48.044 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_committed", , 2020-10-29T09:34:48.044 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2020-10-29T09:34:43.597604+0000", 2020-10-29T22:44:45.070 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:48.057 INFO:teuthology.orchestra.run.smithi115.stdout: }, , 2020-10-29T09:34:48.057 INFO:teuthology.orchestra.run.smithi115.stdout: { , 2020-10-29T09:34:48.057 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2020-10-29T22:44:45.070 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:48.071 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2111000000000003e-05 , 2020-10-29T09:34:48.071 INFO:teuthology.orchestra.run.smithi115.stdout: }, , 2020-10-29T09:34:48.071 INFO:teuthology.orchestra.run.smithi115.stdout: { 2020-10-29T22:44:45.071 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:48.085 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0043861200000000003 , 2020-10-29T09:34:48.085 INFO:teuthology.orchestra.run.smithi115.stdout: } , 2020-10-29T09:34:48.085 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2020-10-29T22:44:45.071 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:48.098 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2020-10-29T09:34:43.624884+0000", , 2020-10-29T09:34:48.098 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0015364280000000001 , 2020-10-29T09:34:48.098 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2020-10-29T22:44:45.071 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:48.112 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", , 2020-10-29T09:34:48.112 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2020-10-29T09:34:43.645473+0000", , 2020-10-29T09:34:48.112 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.897e-05 2020-10-29T22:44:45.072 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:48.125 INFO:teuthology.orchestra.run.smithi115.stdout: { , 2020-10-29T09:34:48.125 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", , 2020-10-29T09:34:48.126 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2020-10-29T09:34:43.651840+0000", 2020-10-29T22:44:45.072 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:48.139 INFO:teuthology.orchestra.run.smithi115.stdout: { , 2020-10-29T09:34:48.139 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", , 2020-10-29T09:34:48.139 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2020-10-29T09:34:44.623676+0000", 2020-10-29T22:44:45.077 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:34:48.152 INFO:teuthology.orchestra.run.smithi115.stdout: }, , 2020-10-29T09:34:48.153 INFO:teuthology.orchestra.run.smithi115.stdout: { , 2020-10-29T09:34:48.153 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2020-10-29T22:44:45.080 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:35:12.599 INFO:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd reweight 2 1 , reweighted osd.2 to 1 (10000) , 2020-10-29T09:35:13.576 INFO:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd primary-affinity 2 1 2020-10-29T22:44:45.081 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:35:32.804 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":630,"fsid":"6c8fb4f3-7344-4c6f-a7ac-4f91a6b86757","created":"2020-10-29T09:24:29.255108+0000","modified":"2020-10-29T09:35:24.953731+0000","last_up_change":"2020-10-29T09:34:39.521463+0000","last_in_change":"2020-10-29T09:35:07.780245+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-10-29T09:24:33.001873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"primary_write_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":21,"pg_placement_num":21,"pg_placement_num_target":21,"pg_num_target":21,"pg_num_pending":21,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"614","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"358","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-10-29T09:24:40.338811+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"primary_write_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":28,"pg_placement_num":28,"pg_placement_num_target":28,"pg_num_target":28,"pg_num_pending":28,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"505","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"479","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2020-10-29T09:24:53.310240+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":3,"min_size":2,"primary_write_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":26,"pg_placement_num":26,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":26,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"611","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"418","auid":0,"snap_mode":"selfmanaged","snap_seq":526,"snap_epoch":598,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"jerasure21profile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"97d05a41-1c42-4233-b4c2-60abf2c27c50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":618,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6809","nonce":13576},{"type":"v1","addr":"172.21.15.173:6811","nonce":13576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":13576},{"type":"v1","addr":"172.21.15.173:6816","nonce":13576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6821","nonce":13576},{"type":"v1","addr":"172.21.15.173:6823","nonce":13576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":13576},{"type":"v1","addr":"172.21.15.173:6820","nonce":13576}]},"public_addr":"172.21.15.173:6811/13576","cluster_addr":"172.21.15.173:6816/13576","heartbeat_back_addr":"172.21.15.173:6823/13576","heartbeat_front_addr":"172.21.15.173:6820/13576","state":["exists","up"]},{"osd":1,"uuid":"2d2ef66b-4738-47e5-85f8-4d2c1efeba46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":617,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":13574},{"type":"v1","addr":"172.21.15.173:6801","nonce":13574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":13574},{"type":"v1","addr":"172.21.15.173:6803","nonce":13574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":13574},{"type":"v1","addr":"172.21.15.173:6807","nonce":13574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":13574},{"type":"v1","addr":"172.21.15.173:6805","nonce":13574}]},"public_addr":"172.21.15.173:6801/13574","cluster_addr":"172.21.15.173:6803/13574","heartbeat_back_addr":"172.21.15.173:6807/13574","heartbeat_front_addr":"172.21.15.173:6805/13574","state":["exists","up"]},{"osd":2,"uuid":"092373a5-85d5-4dea-b8b1-3e90c95ea6e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":617,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":13575},{"type":"v1","addr":"172.21.15.173:6810","nonce":13575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":13575},{"type":"v1","addr":"172.21.15.173:6813","nonce":13575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":13575},{"type":"v1","addr":"172.21.15.173:6822","nonce":13575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6815","nonce":13575},{"type":"v1","addr":"172.21.15.173:6817","nonce":13575}]},"public_addr":"172.21.15.173:6810/13575","cluster_addr":"172.21.15.173:6813/13575","heartbeat_back_addr":"172.21.15.173:6822/13575","heartbeat_front_addr":"172.21.15.173:6817/13575","state":["exists","up"]},{"osd":3,"uuid":"4edc4c1f-0b24-45ca-8e1f-b3ab1c38d47a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":619,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":13605},{"type":"v1","addr":"172.21.15.173:6825","nonce":13605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":13605},{"type":"v1","addr":"172.21.15.173:6827","nonce":13605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":13605},{"type":"v1","addr":"172.21.15.173:6831","nonce":13605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":13605},{"type":"v1","addr":"172.21.15.173:6829","nonce":13605}]},"public_addr":"172.21.15.173:6825/13605","cluster_addr":"172.21.15.173:6827/13605","heartbeat_back_addr":"172.21.15.173:6831/13605","heartbeat_front_addr":"172.21.15.173:6829/13605","state":["exists","up"]},{"osd":4,"uuid":"02e7b054-c0f0-40b9-a31a-d128587532df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":619,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":13692},{"type":"v1","addr":"172.21.15.115:6812","nonce":13692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6815","nonce":13692},{"type":"v1","addr":"172.21.15.115:6818","nonce":13692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":13692},{"type":"v1","addr":"172.21.15.115:6829","nonce":13692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6821","nonce":13692},{"type":"v1","addr":"172.21.15.115:6824","nonce":13692}]},"public_addr":"172.21.15.115:6812/13692","cluster_addr":"172.21.15.115:6818/13692","heartbeat_back_addr":"172.21.15.115:6829/13692","heartbeat_front_addr":"172.21.15.115:6824/13692","state":["exists","up"]},{"osd":5,"uuid":"061e8937-6aff-4d1a-bb5b-da932d4faa2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":561,"up_from":595,"up_thru":616,"down_at":562,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":27311},{"type":"v1","addr":"172.21.15.115:6801","nonce":27311}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":27311},{"type":"v1","addr":"172.21.15.115:6803","nonce":27311}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":27311},{"type":"v1","addr":"172.21.15.115:6807","nonce":27311}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":27311},{"type":"v1","addr":"172.21.15.115:6805","nonce":27311}]},"public_addr":"172.21.15.115:6801/27311","cluster_addr":"172.21.15.115:6803/27311","heartbeat_back_addr":"172.21.15.115:6807/27311","heartbeat_front_addr":"172.21.15.115:6805/27311","state":["exists","up"]},{"osd":6,"uuid":"b1f6af44-fa94-48a1-877e-217a36103497","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":618,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":13693},{"type":"v1","addr":"172.21.15.115:6813","nonce":13693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":13693},{"type":"v1","addr":"172.21.15.115:6819","nonce":13693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":13693},{"type":"v1","addr":"172.21.15.115:6831","nonce":13693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":13693},{"type":"v1","addr":"172.21.15.115:6825","nonce":13693}]},"public_addr":"172.21.15.115:6813/13693","cluster_addr":"172.21.15.115:6819/13693","heartbeat_back_addr":"172.21.15.115:6831/13693","heartbeat_front_addr":"172.21.15.115:6825/13693","state":["exists","up"]},{"osd":7,"uuid":"45a73d9b-b686-4c7b-8001-a3787cda08b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":628,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6809","nonce":13691},{"type":"v1","addr":"172.21.15.115:6811","nonce":13691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":13691},{"type":"v1","addr":"172.21.15.115:6817","nonce":13691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6827","nonce":13691},{"type":"v1","addr":"172.21.15.115:6830","nonce":13691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":13691},{"type":"v1","addr":"172.21.15.115:6823","nonce":13691}]},"public_addr":"172.21.15.115:6811/13691","cluster_addr":"172.21.15.115:6817/13691","heartbeat_back_addr":"172.21.15.115:6830/13691","heartbeat_front_addr":"172.21.15.115:6823/13691","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-29T09:34:38.886952+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-29T09:34:39.808517+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-29T09:34:39.698376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-29T09:34:42.008412+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-29T09:34:40.250426+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-29T09:34:39.800050+0000","dead_epoch":562},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-29T09:34:39.583764+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-29T09:34:39.608873+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.4","osds":[4,7]},{"pgid":"2.6","osds":[4,7]},{"pgid":"3.4","osds":[1,0,2]}],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":4,"to":5}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"jerasure21profile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","name":"jerasure21profile","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} , 2020-10-29T09:35:33.822 INFO:tasks.ceph:Scrubbing osd.0 , 2020-10-29T09:35:33.822 INFO:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.0 config set osd_debug_deep_scrub_sleep 0 2020-10-29T22:44:45.083 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:35:38.230 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2020-10-29T09:35:18.115816+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=9, tm_min=35, tm_sec=18, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=9, tm_min=35, tm_sec=32, tm_wday=3, tm_yday=303, tm_isdst=0) , 2020-10-29T09:35:38.231 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2020-10-29T09:34:36.933765+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=9, tm_min=34, tm_sec=36, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=9, tm_min=35, tm_sec=32, tm_wday=3, tm_yday=303, tm_isdst=0) , 2020-10-29T09:35:38.231 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2020-10-29T09:34:57.579693+0000 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=9, tm_min=34, tm_sec=57, tm_wday=3, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=29, tm_hour=9, tm_min=35, tm_sec=32, tm_wday=3, tm_yday=303, tm_isdst=0) 2020-10-29T22:44:45.083 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:36:19.300 DEBUG:tasks.ceph.osd.0:waiting for process to exit , 2020-10-29T09:36:19.300 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-29T09:36:19.375 INFO:tasks.ceph.osd.0:Stopped 2020-10-29T22:44:45.084 WARNING:scrape:Ignoring malparsed backtrace: 2020-10-29T09:37:57.550 INFO:tasks.ceph:Archiving logs... , 2020-10-29T09:37:57.551 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/log/ceph to /home/teuthworker/archive/shuzhenyi-2020-10-29_07:52:44-rados:thrash-erasure-code-wip-zhenyi-testing-2020-10-28-1530-distro-basic-smithi/5570824/remote/smithi173/log , 2020-10-29T09:37:57.552 INFO:teuthology.orchestra.run.smithi173:> sudo tar cz -f - -C /var/log/ceph -- . 2020-10-29T22:44:45.201 INFO:scrape:Found 4 distinct failure reasons 2020-10-29T22:44:45.202 INFO:scrape:Dead: b'2020-10-29T21:26:57.262 DEBUG:teuthology.exit:Finished running handlers' 2020-10-29T22:44:45.203 INFO:scrape:7 jobs: ['5570848', '5570697', '5570700', '5570774', '5570937', '5570727', '5570807'] 2020-10-29T22:44:45.203 INFO:scrape:suites intersection: ['clusters/{fixed-2', 'openstack}', 'rados', 'rados:thrash-erasure-code/{ceph', 'thrashosds-health'] 2020-10-29T22:44:45.203 INFO:scrape:suites union: ['clusters/{fixed-2', 'fast/fast', 'fast/normal', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados:thrash-erasure-code/{ceph', 'recovery-overrides/{more-active-recovery}', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-async-recovery}', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashers/fastread', 'thrashers/minsize_recovery', 'thrashosds-health', 'workloads/ec-rados-plugin=clay-k=4-m=2}', 'workloads/ec-rados-plugin=jerasure-k=2-m=1}', 'workloads/ec-rados-plugin=jerasure-k=3-m=1}', 'workloads/ec-small-objects-balanced}', 'workloads/ec-small-objects}'] 2020-10-29T22:44:45.204 INFO:scrape: 2020-10-29T22:44:45.204 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' 2020-10-29T22:44:45.204 INFO:scrape:9 jobs: ['5570717', '5570831', '5570805', '5570783', '5570891', '5570909', '5570747', '5570824', '5570993'] 2020-10-29T22:44:45.208 INFO:scrape:suites intersection: ['clusters/{fixed-2', 'openstack}', 'rados', 'rados:thrash-erasure-code/{ceph', 'thrashosds-health'] 2020-10-29T22:44:45.208 INFO:scrape:suites union: ['clusters/{fixed-2', 'fast/fast', 'fast/normal', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados:thrash-erasure-code/{ceph', 'recovery-overrides/{default}', 'recovery-overrides/{more-active-recovery}', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashers/default', 'thrashers/morepggrow', 'thrashers/pggrow', 'thrashosds-health', 'workloads/ec-rados-plugin=clay-k=4-m=2}', 'workloads/ec-rados-plugin=jerasure-k=2-m=1}', 'workloads/ec-rados-plugin=jerasure-k=3-m=1}'] 2020-10-29T22:44:45.209 INFO:scrape: 2020-10-29T22:44:45.209 INFO:scrape:Failure: psutil.NoSuchProcess process no longer exists (pid=2705) 2020-10-29T22:44:45.209 INFO:scrape:2 jobs: ['5570912', '5570711'] 2020-10-29T22:44:45.210 INFO:scrape:suites intersection: ['clusters/{fixed-2', 'openstack}', 'rados', 'rados:thrash-erasure-code/{ceph', 'thrashosds-health'] 2020-10-29T22:44:45.210 INFO:scrape:suites union: ['clusters/{fixed-2', 'fast/fast', 'fast/normal', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/osd-delay', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados:thrash-erasure-code/{ceph', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/fastread', 'thrashers/minsize_recovery', 'thrashosds-health', 'workloads/ec-rados-plugin=jerasure-k=3-m=1}', 'workloads/ec-small-objects}'] 2020-10-29T22:44:45.210 INFO:scrape: 2020-10-29T22:44:45.211 INFO:scrape:Failure: "2020-10-29T08:46:05.623656+0000 mon.b (mon.0) 50 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log 2020-10-29T22:44:45.213 INFO:scrape:2 jobs: ['5570740', '5570884'] 2020-10-29T22:44:45.213 INFO:scrape:suites intersection: ['clusters/{fixed-2', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'openstack}', 'rados', 'rados:thrash-erasure-code/{ceph', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{rhel_8}', 'thrashosds-health', 'workloads/ec-rados-plugin=jerasure-k=2-m=1}'] 2020-10-29T22:44:45.213 INFO:scrape:suites union: ['clusters/{fixed-2', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados:thrash-erasure-code/{ceph', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{rhel_8}', 'thrashers/fastread', 'thrashers/pggrow', 'thrashosds-health', 'workloads/ec-rados-plugin=jerasure-k=2-m=1}'] 2020-10-29T22:44:45.214 INFO:scrape: