2021-01-29T13:00:25.539 INFO:root:teuthology version: 1.0.0-f185165d 2021-01-29T13:00:25.547 INFO:teuthology.results:subset: '976/9999' 2021-01-29T13:00:25.547 INFO:teuthology.results:seed: '2477' 2021-01-29T18:34:32.715 INFO:root:teuthology version: 1.0.0-f185165d 2021-01-29T18:34:32.752 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-01-30T06:13:00.341 INFO:teuthology.results:Tests finished! gathering results... 2021-01-30T06:14:42.425 INFO:scrape:Found 439 jobs 2021-01-30T06:14:43.360 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-297-g27dc1eb3 (27dc1eb32728beb0c69aa65131bc609ef680c0aa) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f5374aa79cd] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f5374aa7ba8] 2021-01-30T06:14:43.362 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-29T16:45:29.791 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-29T16:45:30.090 INFO:teuthology.orchestra.run.smithi087.stdout: 2021-01-30T06:14:43.362 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T16:46:32.264 INFO:teuthology.orchestra.run:waiting for 300 , 2021-01-29T16:46:32.311 INFO:tasks.ceph.mon.a:Stopped , 2021-01-29T16:46:32.312 INFO:tasks.ceph:Checking cluster log for badness... 2021-01-30T06:14:43.363 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T16:50:28.361 INFO:teuthology.task.internal.syslog:Checking logs for errors... , 2021-01-29T16:50:28.363 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi087.front.sepia.ceph.com , 2021-01-29T16:50:28.363 DEBUG:teuthology.orchestra.run.smithi087:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-01-30T06:14:43.475 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-297-g27dc1eb3 (27dc1eb32728beb0c69aa65131bc609ef680c0aa) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f5374aa79cd] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f5374aa7ba8] 2021-01-30T06:14:43.479 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-29T16:45:29.791 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-29T16:45:30.090 INFO:teuthology.orchestra.run.smithi087.stdout: 2021-01-30T06:14:43.480 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T16:46:32.264 INFO:teuthology.orchestra.run:waiting for 300 , 2021-01-29T16:46:32.311 INFO:tasks.ceph.mon.a:Stopped , 2021-01-29T16:46:32.312 INFO:tasks.ceph:Checking cluster log for badness... 2021-01-30T06:14:43.480 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T16:50:28.361 INFO:teuthology.task.internal.syslog:Checking logs for errors... , 2021-01-29T16:50:28.363 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi087.front.sepia.ceph.com , 2021-01-29T16:50:28.363 DEBUG:teuthology.orchestra.run.smithi087:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-01-30T06:14:43.585 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-297-g27dc1eb3 (27dc1eb32728beb0c69aa65131bc609ef680c0aa) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f5374aa79cd] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f5374aa7ba8] 2021-01-30T06:14:43.586 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-29T16:45:29.791 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-29T16:45:30.090 INFO:teuthology.orchestra.run.smithi087.stdout: 2021-01-30T06:14:43.587 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T16:46:32.264 INFO:teuthology.orchestra.run:waiting for 300 , 2021-01-29T16:46:32.311 INFO:tasks.ceph.mon.a:Stopped , 2021-01-29T16:46:32.312 INFO:tasks.ceph:Checking cluster log for badness... 2021-01-30T06:14:43.588 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T16:50:28.361 INFO:teuthology.task.internal.syslog:Checking logs for errors... , 2021-01-29T16:50:28.363 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi087.front.sepia.ceph.com , 2021-01-29T16:50:28.363 DEBUG:teuthology.orchestra.run.smithi087:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-01-30T06:14:43.698 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-297-g27dc1eb3 (27dc1eb32728beb0c69aa65131bc609ef680c0aa) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f5374aa79cd] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f5374aa7ba8] 2021-01-30T06:14:43.700 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-29T16:45:29.791 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-29T16:45:30.090 INFO:teuthology.orchestra.run.smithi087.stdout: 2021-01-30T06:14:43.700 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T16:46:32.264 INFO:teuthology.orchestra.run:waiting for 300 , 2021-01-29T16:46:32.311 INFO:tasks.ceph.mon.a:Stopped , 2021-01-29T16:46:32.312 INFO:tasks.ceph:Checking cluster log for badness... 2021-01-30T06:14:43.701 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T16:50:28.361 INFO:teuthology.task.internal.syslog:Checking logs for errors... , 2021-01-29T16:50:28.363 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi087.front.sepia.ceph.com , 2021-01-29T16:50:28.363 DEBUG:teuthology.orchestra.run.smithi087:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-01-30T06:14:43.816 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-297-g27dc1eb3 (27dc1eb32728beb0c69aa65131bc609ef680c0aa) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f5374aa79cd] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f5374aa7ba8] 2021-01-30T06:14:43.818 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2021-01-29T16:45:29.791 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-29T16:45:30.090 INFO:teuthology.orchestra.run.smithi087.stdout: 2021-01-30T06:14:43.819 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T16:46:32.264 INFO:teuthology.orchestra.run:waiting for 300 , 2021-01-29T16:46:32.311 INFO:tasks.ceph.mon.a:Stopped , 2021-01-29T16:46:32.312 INFO:tasks.ceph:Checking cluster log for badness... 2021-01-30T06:14:43.820 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T16:50:28.361 INFO:teuthology.task.internal.syslog:Checking logs for errors... , 2021-01-29T16:50:28.363 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi087.front.sepia.ceph.com , 2021-01-29T16:50:28.363 DEBUG:teuthology.orchestra.run.smithi087:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-01-30T06:14:56.711 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-297-g27dc1eb3 (27dc1eb32728beb0c69aa65131bc609ef680c0aa) quincy (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7fde757ed15a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x55bfbfa77fe6] 2021-01-30T06:14:56.716 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T15:58:20.598 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. , 2021-01-29T15:58:20.608 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops , 2021-01-29T15:58:20.786 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2021-01-30T06:14:56.718 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T15:58:43.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 , 2021-01-29T15:58:43.760 INFO:teuthology.orchestra.run.smithi067.stdout:64424509480 , 2021-01-29T15:58:43.773 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509480 got 64424509480 for osd.2 2021-01-30T06:14:56.719 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T15:58:52.402 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2021-01-29T15:58:35.676485+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=29, tm_hour=15, tm_min=58, tm_sec=35, tm_wday=4, tm_yday=29, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=29, tm_hour=15, tm_min=58, tm_sec=47, tm_wday=4, tm_yday=29, tm_isdst=0) , 2021-01-29T15:58:52.402 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2021-01-29T15:58:27.421583+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=29, tm_hour=15, tm_min=58, tm_sec=27, tm_wday=4, tm_yday=29, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=29, tm_hour=15, tm_min=58, tm_sec=47, tm_wday=4, tm_yday=29, tm_isdst=0) , 2021-01-29T15:58:52.403 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2021-01-29T15:57:43.068254+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=29, tm_hour=15, tm_min=57, tm_sec=43, tm_wday=4, tm_yday=29, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=29, tm_hour=15, tm_min=58, tm_sec=47, tm_wday=4, tm_yday=29, tm_isdst=0) 2021-01-30T06:14:56.720 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T15:59:33.947 INFO:tasks.ceph.mgr.x:Stopped , 2021-01-29T15:59:33.948 INFO:teuthology.misc:Shutting down mon daemons... , 2021-01-29T15:59:33.948 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2021-01-30T06:14:56.720 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T16:00:36.936 DEBUG:teuthology.orchestra.run.smithi205:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits , 2021-01-29T16:00:37.017 DEBUG:teuthology.run_tasks:Unwinding manager clock , 2021-01-29T16:00:37.030 INFO:teuthology.task.clock:Checking final clock skew... 2021-01-30T06:14:56.721 WARNING:scrape:Ignoring malparsed backtrace: IndexError: list index out of range , 2021-01-29T16:01:02.075 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload , 2021-01-29T16:01:02.089 INFO:teuthology.task.internal:Not uploading archives. 2021-01-30T06:14:56.881 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-297-g27dc1eb3 (27dc1eb32728beb0c69aa65131bc609ef680c0aa) quincy (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7fde757ed15a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x55bfbfa77fe6] 2021-01-30T06:14:56.883 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T15:58:20.598 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. , 2021-01-29T15:58:20.608 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops , 2021-01-29T15:58:20.786 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2021-01-30T06:14:56.885 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T15:58:43.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 , 2021-01-29T15:58:43.760 INFO:teuthology.orchestra.run.smithi067.stdout:64424509480 , 2021-01-29T15:58:43.773 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509480 got 64424509480 for osd.2 2021-01-30T06:14:56.886 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T15:58:52.402 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2021-01-29T15:58:35.676485+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=29, tm_hour=15, tm_min=58, tm_sec=35, tm_wday=4, tm_yday=29, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=29, tm_hour=15, tm_min=58, tm_sec=47, tm_wday=4, tm_yday=29, tm_isdst=0) , 2021-01-29T15:58:52.402 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2021-01-29T15:58:27.421583+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=29, tm_hour=15, tm_min=58, tm_sec=27, tm_wday=4, tm_yday=29, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=29, tm_hour=15, tm_min=58, tm_sec=47, tm_wday=4, tm_yday=29, tm_isdst=0) , 2021-01-29T15:58:52.403 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2021-01-29T15:57:43.068254+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=29, tm_hour=15, tm_min=57, tm_sec=43, tm_wday=4, tm_yday=29, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=29, tm_hour=15, tm_min=58, tm_sec=47, tm_wday=4, tm_yday=29, tm_isdst=0) 2021-01-30T06:14:56.886 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T15:59:33.947 INFO:tasks.ceph.mgr.x:Stopped , 2021-01-29T15:59:33.948 INFO:teuthology.misc:Shutting down mon daemons... , 2021-01-29T15:59:33.948 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2021-01-30T06:14:56.887 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T16:00:36.936 DEBUG:teuthology.orchestra.run.smithi205:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits , 2021-01-29T16:00:37.017 DEBUG:teuthology.run_tasks:Unwinding manager clock , 2021-01-29T16:00:37.030 INFO:teuthology.task.clock:Checking final clock skew... 2021-01-30T06:14:56.888 WARNING:scrape:Ignoring malparsed backtrace: IndexError: list index out of range , 2021-01-29T16:01:02.075 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload , 2021-01-29T16:01:02.089 INFO:teuthology.task.internal:Not uploading archives. 2021-01-30T06:14:57.030 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-297-g27dc1eb3 (27dc1eb32728beb0c69aa65131bc609ef680c0aa) quincy (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7fde757ed15a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x55bfbfa77fe6] 2021-01-30T06:14:57.031 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T15:58:20.598 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. , 2021-01-29T15:58:20.608 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops , 2021-01-29T15:58:20.786 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2021-01-30T06:14:57.033 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T15:58:43.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 , 2021-01-29T15:58:43.760 INFO:teuthology.orchestra.run.smithi067.stdout:64424509480 , 2021-01-29T15:58:43.773 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509480 got 64424509480 for osd.2 2021-01-30T06:14:57.034 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T15:58:52.402 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2021-01-29T15:58:35.676485+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=29, tm_hour=15, tm_min=58, tm_sec=35, tm_wday=4, tm_yday=29, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=29, tm_hour=15, tm_min=58, tm_sec=47, tm_wday=4, tm_yday=29, tm_isdst=0) , 2021-01-29T15:58:52.402 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2021-01-29T15:58:27.421583+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=29, tm_hour=15, tm_min=58, tm_sec=27, tm_wday=4, tm_yday=29, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=29, tm_hour=15, tm_min=58, tm_sec=47, tm_wday=4, tm_yday=29, tm_isdst=0) , 2021-01-29T15:58:52.403 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2021-01-29T15:57:43.068254+0000 time.struct_time(tm_year=2021, tm_mon=1, tm_mday=29, tm_hour=15, tm_min=57, tm_sec=43, tm_wday=4, tm_yday=29, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=1, tm_mday=29, tm_hour=15, tm_min=58, tm_sec=47, tm_wday=4, tm_yday=29, tm_isdst=0) 2021-01-30T06:14:57.034 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T15:59:33.947 INFO:tasks.ceph.mgr.x:Stopped , 2021-01-29T15:59:33.948 INFO:teuthology.misc:Shutting down mon daemons... , 2021-01-29T15:59:33.948 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2021-01-30T06:14:57.035 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T16:00:36.936 DEBUG:teuthology.orchestra.run.smithi205:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits , 2021-01-29T16:00:37.017 DEBUG:teuthology.run_tasks:Unwinding manager clock , 2021-01-29T16:00:37.030 INFO:teuthology.task.clock:Checking final clock skew... 2021-01-30T06:14:57.036 WARNING:scrape:Ignoring malparsed backtrace: IndexError: list index out of range , 2021-01-29T16:01:02.075 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload , 2021-01-29T16:01:02.089 INFO:teuthology.task.internal:Not uploading archives. 2021-01-30T06:14:57.482 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-297-g27dc1eb3 (27dc1eb32728beb0c69aa65131bc609ef680c0aa) quincy (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f30135d315a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x55fd1cf11fe6] 2021-01-30T06:14:57.484 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T13:55:49.136 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument , 2021-01-29T13:55:49.136 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. , 2021-01-29T13:55:49.151 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2021-01-30T06:14:57.484 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T13:56:38.516 INFO:teuthology.orchestra.run.smithi183.stdout:oload 1000 , 2021-01-29T13:56:38.516 INFO:teuthology.orchestra.run.smithi183.stdout:max_change 3 , 2021-01-29T13:56:38.517 INFO:teuthology.orchestra.run.smithi183.stdout:max_change_osds 4 2021-01-30T06:14:57.486 WARNING:scrape:Ignoring malparsed backtrace: set osd.3 primary-affinity to 1 (8655362) , 2021-01-29T13:56:51.495 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 , 2021-01-29T13:56:52.480+0000 7f8f78ab0700 -1 osd.2 132 handle_osd_map injecting map crc failure 2021-01-30T06:14:57.491 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T13:57:08.429 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":278,"stamp":"2021-01-29T13:57:06.667864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20305121604,"num_objects":6822,"num_object_clones":0,"num_object_copies":19368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2076,"num_objects_unfound":0,"num_objects_dirty":4824,"num_whiteouts":0,"num_read":9651,"num_read_kb":4830,"num_write":25509,"num_write_kb":14727220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2477,"num_bytes_recovered":5602499186,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":21,"num_bytes_hit_set_archive":12196,"num_flush":2862,"num_flush_kb":8348668,"num_evict":884,"num_evict_kb":2571590,"num_promote":4835,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8249,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15129,"ondisk_log_size":15129,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":45,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":24946968,"kb_used_data":24625140,"kb_used_omap":1,"kb_used_meta":321598,"kb_avail":730027752,"statfs":{"total":773094113280,"available":747548418048,"internally_reserved":0,"allocated":25216143360,"data_stored":30051268153,"data_compressed":504955484,"data_compressed_allocated":4927905792,"data_compressed_original":9855811584,"omap_allocated":1944,"internal_metadata":329316456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-170,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":394035071,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8249,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.133172"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":"145","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.847683+0000","last_change":"2021-01-29T13:55:57.162929+0000","last_active":"2021-01-29T13:57:00.847683+0000","last_peered":"2021-01-29T13:57:00.847683+0000","last_clean":"2021-01-29T13:57:00.847683+0000","last_became_active":"2021-01-29T13:52:24.188931+0000","last_became_peered":"2021-01-29T13:52:24.188931+0000","last_unstale":"2021-01-29T13:57:00.847683+0000","last_undegraded":"2021-01-29T13:57:00.847683+0000","last_fullsized":"2021-01-29T13:57:00.847683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:55:57.162808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:55:57.162808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"120'6449","reported_seq":"8595","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.652824+0000","last_change":"2021-01-29T13:55:45.625490+0000","last_active":"2021-01-29T13:57:00.652824+0000","last_peered":"2021-01-29T13:57:00.652824+0000","last_clean":"2021-01-29T13:57:00.652824+0000","last_became_active":"2021-01-29T13:52:39.830011+0000","last_became_peered":"2021-01-29T13:52:39.830011+0000","last_unstale":"2021-01-29T13:57:00.652824+0000","last_undegraded":"2021-01-29T13:57:00.652824+0000","last_fullsized":"2021-01-29T13:57:00.652824+0000","mapping_epoch":26,"log_start":"72'3403","ondisk_log_start":"72'3403","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'6250","last_scrub_stamp":"2021-01-29T13:55:45.625379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:38.811171+0000","last_clean_scrub_stamp":"2021-01-29T13:55:45.625379+0000","log_size":3046,"ondisk_log_size":3046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2928382292,"num_objects":986,"num_object_clones":0,"num_object_copies":1972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":2501,"num_read_kb":1252,"num_write":5879,"num_write_kb":1639641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":3315,"num_flush":761,"num_flush_kb":2192275,"num_evict":269,"num_evict_kb":760560,"num_promote":1249,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"112","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.552366+0000","last_change":"2021-01-29T13:55:42.661441+0000","last_active":"2021-01-29T13:57:00.552366+0000","last_peered":"2021-01-29T13:57:00.552366+0000","last_clean":"2021-01-29T13:57:00.552366+0000","last_became_active":"2021-01-29T13:53:29.493255+0000","last_became_peered":"2021-01-29T13:53:29.493255+0000","last_unstale":"2021-01-29T13:57:00.552366+0000","last_undegraded":"2021-01-29T13:57:00.552366+0000","last_fullsized":"2021-01-29T13:57:00.552366+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:55:42.661371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:55:42.661371+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3","version":"120'671","reported_seq":"1016","reported_epoch":"142","state":"active+remapped+backfilling","last_fresh":"2021-01-29T13:57:05.597246+0000","last_change":"2021-01-29T13:56:47.680232+0000","last_active":"2021-01-29T13:57:05.597246+0000","last_peered":"2021-01-29T13:57:05.597246+0000","last_clean":"2021-01-29T13:56:36.386610+0000","last_became_active":"2021-01-29T13:56:47.674484+0000","last_became_peered":"2021-01-29T13:56:47.674484+0000","last_unstale":"2021-01-29T13:57:05.597246+0000","last_undegraded":"2021-01-29T13:57:05.597246+0000","last_fullsized":"2021-01-29T13:57:05.597246+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'570","last_scrub_stamp":"2021-01-29T13:55:44.911108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:35.792587+0000","last_clean_scrub_stamp":"2021-01-29T13:55:44.911108+0000","log_size":671,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2038945493,"num_objects":671,"num_object_clones":0,"num_object_copies":2684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":590,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":671,"num_write_kb":1991482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":601,"num_bytes_recovered":1847436236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4,7],"acting":[6,2,4,7],"avail_no_missing":["6(0)","2(1)","4(2)","7(3)"],"object_location_counts":[{"shards":"2(1),4(2),6(0),7(3)","objects":671}],"blocked_by":[],"up_primary":0,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"142","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:01.550240+0000","last_change":"2021-01-29T13:57:01.550191+0000","last_active":"2021-01-29T13:57:01.550240+0000","last_peered":"2021-01-29T13:57:01.550240+0000","last_clean":"2021-01-29T13:57:01.550240+0000","last_became_active":"2021-01-29T13:52:23.765595+0000","last_became_peered":"2021-01-29T13:52:23.765595+0000","last_unstale":"2021-01-29T13:57:01.550240+0000","last_undegraded":"2021-01-29T13:57:01.550240+0000","last_fullsized":"2021-01-29T13:57:01.550240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:57:01.550087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:57:01.550087+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"117'710","reported_seq":"880","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.847676+0000","last_change":"2021-01-29T13:56:09.328006+0000","last_active":"2021-01-29T13:57:00.847676+0000","last_peered":"2021-01-29T13:57:00.847676+0000","last_clean":"2021-01-29T13:57:00.847676+0000","last_became_active":"2021-01-29T13:53:09.969063+0000","last_became_peered":"2021-01-29T13:53:09.969063+0000","last_unstale":"2021-01-29T13:57:00.847676+0000","last_undegraded":"2021-01-29T13:57:00.847676+0000","last_fullsized":"2021-01-29T13:57:00.847676+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'710","last_scrub_stamp":"2021-01-29T13:56:09.327903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:35.792587+0000","last_clean_scrub_stamp":"2021-01-29T13:56:09.327903+0000","log_size":710,"ondisk_log_size":710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2107977089,"num_objects":710,"num_object_clones":0,"num_object_copies":2840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":710,"num_write_kb":2058922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2,4],"acting":[5,7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"155","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.849258+0000","last_change":"2021-01-29T13:56:59.467725+0000","last_active":"2021-01-29T13:57:00.849258+0000","last_peered":"2021-01-29T13:57:00.849258+0000","last_clean":"2021-01-29T13:57:00.849258+0000","last_became_active":"2021-01-29T13:52:24.188578+0000","last_became_peered":"2021-01-29T13:52:24.188578+0000","last_unstale":"2021-01-29T13:57:00.849258+0000","last_undegraded":"2021-01-29T13:57:00.849258+0000","last_fullsized":"2021-01-29T13:57:00.849258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:56:59.467654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:56:59.467654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"142'4","reported_seq":"152","reported_epoch":"142","state":"active+clean","last_fresh":"2021-01-29T13:57:01.982145+0000","last_change":"2021-01-29T13:57:01.982145+0000","last_active":"2021-01-29T13:57:01.982145+0000","last_peered":"2021-01-29T13:57:01.982145+0000","last_clean":"2021-01-29T13:57:01.982145+0000","last_became_active":"2021-01-29T13:52:20.655015+0000","last_became_peered":"2021-01-29T13:52:20.655015+0000","last_unstale":"2021-01-29T13:57:01.982145+0000","last_undegraded":"2021-01-29T13:57:01.982145+0000","last_fullsized":"2021-01-29T13:57:01.982145+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'4","last_scrub_stamp":"2021-01-29T13:57:01.982073+0000","last_deep_scrub":"142'4","last_deep_scrub_stamp":"2021-01-29T13:57:01.982073+0000","last_clean_scrub_stamp":"2021-01-29T13:57:01.982073+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8249,"num_omap_keys":2,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"120'761","reported_seq":"2454","reported_epoch":"142","state":"active+remapped+backfilling","last_fresh":"2021-01-29T13:57:03.004043+0000","last_change":"2021-01-29T13:56:48.363777+0000","last_active":"2021-01-29T13:57:03.004043+0000","last_peered":"2021-01-29T13:57:03.004043+0000","last_clean":"2021-01-29T13:56:36.386422+0000","last_became_active":"2021-01-29T13:56:48.362095+0000","last_became_peered":"2021-01-29T13:56:48.362095+0000","last_unstale":"2021-01-29T13:57:03.004043+0000","last_undegraded":"2021-01-29T13:57:03.004043+0000","last_fullsized":"2021-01-29T13:57:03.004043+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'761","last_scrub_stamp":"2021-01-29T13:55:53.981049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:35.792587+0000","last_clean_scrub_stamp":"2021-01-29T13:55:53.981049+0000","log_size":761,"ondisk_log_size":761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2244508217,"num_objects":761,"num_object_clones":0,"num_object_copies":3044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":637,"num_objects_unfound":0,"num_objects_dirty":761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":761,"num_write_kb":2192275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":747,"num_bytes_recovered":2213000897,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7,0],"acting":[4,2,7,1],"avail_no_missing":["4(0)","1(3)","2(1)","7(2)"],"object_location_counts":[{"shards":"1(3),2(1),4(0),7(2)","objects":761}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"144","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.849237+0000","last_change":"2021-01-29T13:56:41.371116+0000","last_active":"2021-01-29T13:57:00.849237+0000","last_peered":"2021-01-29T13:57:00.849237+0000","last_clean":"2021-01-29T13:57:00.849237+0000","last_became_active":"2021-01-29T13:52:23.765671+0000","last_became_peered":"2021-01-29T13:52:23.765671+0000","last_unstale":"2021-01-29T13:57:00.849237+0000","last_undegraded":"2021-01-29T13:57:00.849237+0000","last_fullsized":"2021-01-29T13:57:00.849237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:56:41.371025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:56:41.371025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"118'720","reported_seq":"886","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.652827+0000","last_change":"2021-01-29T13:55:59.202455+0000","last_active":"2021-01-29T13:57:00.652827+0000","last_peered":"2021-01-29T13:57:00.652827+0000","last_clean":"2021-01-29T13:57:00.652827+0000","last_became_active":"2021-01-29T13:53:09.587252+0000","last_became_peered":"2021-01-29T13:53:09.587252+0000","last_unstale":"2021-01-29T13:57:00.652827+0000","last_undegraded":"2021-01-29T13:57:00.652827+0000","last_fullsized":"2021-01-29T13:57:00.652827+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'720","last_scrub_stamp":"2021-01-29T13:55:59.202349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:35.792587+0000","last_clean_scrub_stamp":"2021-01-29T13:55:59.202349+0000","log_size":720,"ondisk_log_size":720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2156154079,"num_objects":720,"num_object_clones":0,"num_object_copies":2880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":720,"num_write_kb":2105989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4,3],"acting":[6,2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"120'5986","reported_seq":"10252","reported_epoch":"142","state":"active+remapped+backfilling","last_fresh":"2021-01-29T13:57:05.668638+0000","last_change":"2021-01-29T13:56:47.680268+0000","last_active":"2021-01-29T13:57:05.668638+0000","last_peered":"2021-01-29T13:57:05.668638+0000","last_clean":"2021-01-29T13:56:36.386614+0000","last_became_active":"2021-01-29T13:56:47.673885+0000","last_became_peered":"2021-01-29T13:56:47.673885+0000","last_unstale":"2021-01-29T13:57:05.668638+0000","last_undegraded":"2021-01-29T13:57:05.668638+0000","last_fullsized":"2021-01-29T13:57:05.668638+0000","mapping_epoch":127,"log_start":"71'2904","ondisk_log_start":"71'2904","created":26,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'5955","last_scrub_stamp":"2021-01-29T13:55:51.275882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:38.811171+0000","last_clean_scrub_stamp":"2021-01-29T13:55:51.275882+0000","log_size":3082,"ondisk_log_size":3082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2947693900,"num_objects":985,"num_object_clones":0,"num_object_copies":1970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":849,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":2316,"num_read_kb":1158,"num_write":5435,"num_write_kb":1544566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1129,"num_bytes_recovered":1542062053,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":2771,"num_flush":671,"num_flush_kb":1991482,"num_evict":178,"num_evict_kb":526959,"num_promote":1172,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":985}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"157","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:01.451085+0000","last_change":"2021-01-29T13:57:01.451085+0000","last_active":"2021-01-29T13:57:01.451085+0000","last_peered":"2021-01-29T13:57:01.451085+0000","last_clean":"2021-01-29T13:57:01.451085+0000","last_became_active":"2021-01-29T13:53:09.377717+0000","last_became_peered":"2021-01-29T13:53:09.377717+0000","last_unstale":"2021-01-29T13:57:01.451085+0000","last_undegraded":"2021-01-29T13:57:01.451085+0000","last_fullsized":"2021-01-29T13:57:01.451085+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:55:33.131622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:55:33.131622+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"140","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.847720+0000","last_change":"2021-01-29T13:56:53.807344+0000","last_active":"2021-01-29T13:57:00.847720+0000","last_peered":"2021-01-29T13:57:00.847720+0000","last_clean":"2021-01-29T13:57:00.847720+0000","last_became_active":"2021-01-29T13:52:40.828334+0000","last_became_peered":"2021-01-29T13:52:40.828334+0000","last_unstale":"2021-01-29T13:57:00.847720+0000","last_undegraded":"2021-01-29T13:57:00.847720+0000","last_fullsized":"2021-01-29T13:57:00.847720+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:56:53.807251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:56:53.807251+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"117'6273","reported_seq":"8319","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.552349+0000","last_change":"2021-01-29T13:55:55.445357+0000","last_active":"2021-01-29T13:57:00.552349+0000","last_peered":"2021-01-29T13:57:00.552349+0000","last_clean":"2021-01-29T13:57:00.552349+0000","last_became_active":"2021-01-29T13:53:09.369472+0000","last_became_peered":"2021-01-29T13:53:09.369472+0000","last_unstale":"2021-01-29T13:57:00.552349+0000","last_undegraded":"2021-01-29T13:57:00.552349+0000","last_fullsized":"2021-01-29T13:57:00.552349+0000","mapping_epoch":42,"log_start":"71'3203","ondisk_log_start":"71'3203","created":26,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'6273","last_scrub_stamp":"2021-01-29T13:55:55.445229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:38.811171+0000","last_clean_scrub_stamp":"2021-01-29T13:55:55.445229+0000","log_size":3070,"ondisk_log_size":3070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2935339657,"num_objects":992,"num_object_clones":0,"num_object_copies":1984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":496,"num_whiteouts":0,"num_read":2414,"num_read_kb":1208,"num_write":5647,"num_write_kb":1593121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":3044,"num_flush":710,"num_flush_kb":2058922,"num_evict":219,"num_evict_kb":640347,"num_promote":1206,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"24","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.611252+0000","last_change":"2021-01-29T13:56:46.442031+0000","last_active":"2021-01-29T13:57:00.611252+0000","last_peered":"2021-01-29T13:57:00.611252+0000","last_clean":"2021-01-29T13:57:00.611252+0000","last_became_active":"2021-01-29T13:56:46.441589+0000","last_became_peered":"2021-01-29T13:56:46.441589+0000","last_unstale":"2021-01-29T13:57:00.611252+0000","last_undegraded":"2021-01-29T13:57:00.611252+0000","last_fullsized":"2021-01-29T13:57:00.611252+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:55:55.134762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:55:55.134762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"118'6270","reported_seq":"8351","reported_epoch":"142","state":"active+clean","last_fresh":"2021-01-29T13:57:04.235378+0000","last_change":"2021-01-29T13:57:04.235378+0000","last_active":"2021-01-29T13:57:04.235378+0000","last_peered":"2021-01-29T13:57:04.235378+0000","last_clean":"2021-01-29T13:57:04.235378+0000","last_became_active":"2021-01-29T13:52:39.823642+0000","last_became_peered":"2021-01-29T13:52:39.823642+0000","last_unstale":"2021-01-29T13:57:04.235378+0000","last_undegraded":"2021-01-29T13:57:04.235378+0000","last_fullsized":"2021-01-29T13:57:04.235378+0000","mapping_epoch":26,"log_start":"71'3205","ondisk_log_start":"71'3205","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6270","last_scrub_stamp":"2021-01-29T13:57:04.235268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:38.811171+0000","last_clean_scrub_stamp":"2021-01-29T13:57:04.235268+0000","log_size":3065,"ondisk_log_size":3065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2946120877,"num_objects":995,"num_object_clones":0,"num_object_copies":1990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":2420,"num_read_kb":1212,"num_write":5684,"num_write_kb":1601214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":3066,"num_flush":720,"num_flush_kb":2105989,"num_evict":218,"num_evict_kb":643724,"num_promote":1208,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":4,"stat_sum":{"num_bytes":11757536726,"num_objects":3958,"num_object_clones":0,"num_object_copies":7916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":849,"num_objects_unfound":0,"num_objects_dirty":1960,"num_whiteouts":0,"num_read":9651,"num_read_kb":4830,"num_write":22645,"num_write_kb":6378542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1129,"num_bytes_recovered":1542062053,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":21,"num_bytes_hit_set_archive":12196,"num_flush":2862,"num_flush_kb":8348668,"num_evict":884,"num_evict_kb":2571590,"num_promote":4835,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11918577664,"data_stored":11828415316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12263,"ondisk_log_size":12263,"up":8,"acting":8,"num_store_stats":6},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":8547584878,"num_objects":2862,"num_object_clones":0,"num_object_copies":11448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1227,"num_objects_unfound":0,"num_objects_dirty":2862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2862,"num_write_kb":8348668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1348,"num_bytes_recovered":4060437133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13299490816,"data_stored":18228555776,"data_compressed":505072361,"data_compressed_allocated":4929064960,"data_compressed_original":9858129920,"omap_allocated":0,"internal_metadata":0},"log_size":2862,"ondisk_log_size":2862,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8249,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":0,"up_from":124,"seq":532575944712,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1942776,"kb_used_data":1921328,"kb_used_omap":0,"kb_used_meta":21439,"kb_avail":92429064,"statfs":{"total":96636764160,"available":94647361536,"internally_reserved":0,"allocated":1967439872,"data_stored":2257930354,"data_compressed":30754208,"data_compressed_allocated":299577344,"data_compressed_original":599154688,"omap_allocated":972,"internal_metadata":21953588},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.23}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98799999999999999}]}]},{"osd":7,"up_from":15,"seq":64424509501,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2311348,"kb_used_data":2281004,"kb_used_omap":0,"kb_used_meta":30336,"kb_avail":92060492,"statfs":{"total":96636764160,"available":94269943808,"internally_reserved":0,"allocated":2335748096,"data_stored":3200465622,"data_compressed":88587048,"data_compressed_allocated":865042432,"data_compressed_original":1730084864,"omap_allocated":0,"internal_metadata":31064064},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:54:26 2021","interfaces":[{"interface":"back","average":{"1min":51.597999999999999,"5min":29.779,"15min":26.143000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":906.00800000000004,"5min":906.00800000000004,"15min":906.00800000000004},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.1040000000000001,"5min":0.95499999999999996,"15min":0.93000000000000005},"min":{"1min":0.61499999999999999,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":4.0259999999999998,"5min":4.0259999999999998,"15min":4.0259999999999998},"last":1.3089999999999999}]},{"osd":1,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.3340000000000001,"15min":1.49},"min":{"1min":0.65800000000000003,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.4410000000000001,"5min":11.195,"15min":11.195},"last":1.272},{"interface":"front","average":{"1min":1.006,"5min":23.977,"15min":32.523000000000003},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.3759999999999999,"5min":777.02499999999998,"15min":777.02499999999998},"last":1.581}]},{"osd":2,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":46.633000000000003,"5min":10.444000000000001,"15min":4.5460000000000003},"min":{"1min":0.62,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":512.423,"5min":512.423,"15min":512.423},"last":0.999},{"interface":"front","average":{"1min":1.1559999999999999,"5min":2.1200000000000001,"15min":1.774},"min":{"1min":0.79300000000000004,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.472,"5min":60.529000000000003,"15min":60.529000000000003},"last":1.3759999999999999}]},{"osd":3,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":10.481,"5min":3.0230000000000001,"15min":1.746},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":211.47900000000001,"5min":211.47900000000001,"15min":211.47900000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.46,"15min":1.607},"min":{"1min":0.92300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.4930000000000001,"5min":11.146000000000001,"15min":11.146000000000001},"last":1.4079999999999999}]},{"osd":4,"last update":"Fri Jan 29 13:56:16 2021","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.264,"15min":1.3380000000000001},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.4299999999999999,"5min":5.8449999999999998,"15min":5.8449999999999998},"last":1.343},{"interface":"front","average":{"1min":1.0229999999999999,"5min":14.401999999999999,"15min":20.280000000000001},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.4399999999999999,"5min":391.66699999999997,"15min":391.66699999999997},"last":1.54}]},{"osd":5,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":3.4470000000000001,"15min":2.2829999999999999},"min":{"1min":0.74199999999999999,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.601,"5min":175.83199999999999,"15min":175.83199999999999},"last":1.625},{"interface":"front","average":{"1min":1.105,"5min":1.2549999999999999,"15min":1.2},"min":{"1min":0.77600000000000002,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.526,"5min":4.4020000000000001,"15min":4.4020000000000001},"last":1.6020000000000001}]},{"osd":6,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.5649999999999999,"15min":1.536},"min":{"1min":0.84199999999999997,"5min":0.754,"15min":0.754},"max":{"1min":1.5269999999999999,"5min":10.063000000000001,"15min":10.063000000000001},"last":1.647},{"interface":"front","average":{"1min":1.123,"5min":14.808,"15min":27.946000000000002},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.4690000000000001,"5min":623.01999999999998,"15min":623.01999999999998},"last":1.754}]}]},{"osd":2,"up_from":15,"seq":64424509500,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5736668,"kb_used_data":5682580,"kb_used_omap":0,"kb_used_meta":54080,"kb_avail":88635172,"statfs":{"total":96636764160,"available":90762416128,"internally_reserved":0,"allocated":5818961920,"data_stored":6956451332,"data_compressed":118712623,"data_compressed_allocated":1158184960,"data_compressed_original":2316369920,"omap_allocated":0,"internal_metadata":55377920},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:54:22 2021","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":29.140000000000001,"15min":33.799999999999997},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":2.012,"5min":565.52099999999996,"15min":565.52099999999996},"last":0.997},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.0209999999999999,"15min":0.94299999999999995},"min":{"1min":0.72499999999999998,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":3.6030000000000002,"5min":3.6030000000000002,"15min":3.6030000000000002},"last":0.77000000000000002}]},{"osd":1,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":40.582999999999998,"5min":17.475000000000001,"15min":6.4610000000000003},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":676.39099999999996,"5min":721.11000000000001,"15min":721.11000000000001},"last":1.0489999999999999},{"interface":"front","average":{"1min":0.89700000000000002,"5min":21.452999999999999,"15min":7.8179999999999996},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.1479999999999999,"5min":1000.376,"15min":1000.376},"last":0.81299999999999994}]},{"osd":3,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":51.539000000000001,"5min":11.381,"15min":4.3440000000000003},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":501.61200000000002,"5min":501.61200000000002,"15min":501.61200000000002},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.85399999999999998,"5min":1.1890000000000001,"15min":1.0640000000000001},"min":{"1min":0.57399999999999995,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.6839999999999999,"5min":6.1870000000000003,"15min":6.1870000000000003},"last":0.78900000000000003}]},{"osd":4,"last update":"Fri Jan 29 13:56:15 2021","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.478,"15min":1.53},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.6719999999999999,"5min":3.694,"15min":3.694},"last":1.6160000000000001},{"interface":"front","average":{"1min":1.155,"5min":1.407,"15min":1.3859999999999999},"min":{"1min":0.871,"5min":0.871,"15min":0.871},"max":{"1min":1.7509999999999999,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":1.6479999999999999}]},{"osd":5,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":21.327999999999999,"5min":5.8109999999999999,"15min":2.7749999999999999},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":183.761,"5min":183.761,"15min":183.761},"last":0.86199999999999999},{"interface":"front","average":{"1min":41.545999999999999,"5min":9.516,"15min":4.0499999999999998},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":521.02499999999998,"5min":521.02499999999998,"15min":521.02499999999998},"last":1.0289999999999999}]},{"osd":6,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.3720000000000001,"15min":1.3009999999999999},"min":{"1min":0.89900000000000002,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.474,"5min":3.46,"15min":3.46},"last":1.544},{"interface":"front","average":{"1min":1.111,"5min":1.4450000000000001,"15min":1.2809999999999999},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.4570000000000001,"5min":7.0209999999999999,"15min":7.0209999999999999},"last":0.88200000000000001}]},{"osd":7,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":8.5039999999999996,"15min":3.754},"min":{"1min":0.99099999999999999,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":1.7050000000000001,"5min":591.18899999999996,"15min":591.18899999999996},"last":1.5820000000000001},{"interface":"front","average":{"1min":52.493000000000002,"5min":11.779999999999999,"15min":4.8220000000000001},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":923.40099999999995,"5min":923.40099999999995,"15min":923.40099999999995},"last":0.95499999999999996}]}]},{"osd":1,"up_from":15,"seq":64424509501,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2154844,"kb_used_data":2115092,"kb_used_omap":0,"kb_used_meta":39744,"kb_avail":92216996,"statfs":{"total":96636764160,"available":94430203904,"internally_reserved":0,"allocated":2165854208,"data_stored":2457969146,"data_compressed":31021897,"data_compressed_allocated":302632960,"data_compressed_original":605265920,"omap_allocated":0,"internal_metadata":40697856},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:54:25 2021","interfaces":[{"interface":"back","average":{"1min":1.7629999999999999,"5min":1.0349999999999999,"15min":0.91300000000000003},"min":{"1min":0.63300000000000001,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":5.0629999999999997,"5min":5.0629999999999997,"15min":5.0629999999999997},"last":0.92400000000000004},{"interface":"front","average":{"1min":39.090000000000003,"5min":8.5060000000000002,"15min":3.4079999999999999},"min":{"1min":0.58699999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":752.73400000000004,"5min":752.73400000000004,"15min":752.73400000000004},"last":0.54300000000000004}]},{"osd":2,"last update":"Fri Jan 29 13:56:29 2021","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":22.648,"15min":29.776},"min":{"1min":0.55100000000000005,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":4.2629999999999999,"5min":832.21600000000001,"15min":832.21600000000001},"last":0.83499999999999996},{"interface":"front","average":{"1min":5.2699999999999996,"5min":11.266,"15min":4.4199999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":84.277000000000001,"5min":899.43399999999997,"15min":899.43399999999997},"last":0.86599999999999999}]},{"osd":3,"last update":"Fri Jan 29 13:56:29 2021","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":1.306,"15min":1.099},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":2.1499999999999999,"5min":6.0579999999999998,"15min":6.0579999999999998},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":5.548,"15min":9.8919999999999995},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.5489999999999999,"5min":244.60900000000001,"15min":244.60900000000001},"last":0.68899999999999995}]},{"osd":4,"last update":"Fri Jan 29 13:56:15 2021","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.4610000000000001,"15min":1.633},"min":{"1min":0.63800000000000001,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.5960000000000001,"5min":5.242,"15min":5.242},"last":0.71399999999999997},{"interface":"front","average":{"1min":1.202,"5min":1.9259999999999999,"15min":2.1680000000000001},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.5760000000000001,"5min":6.2590000000000003,"15min":6.2590000000000003},"last":1.024}]},{"osd":5,"last update":"Fri Jan 29 13:56:29 2021","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":14.667999999999999,"15min":5.694},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":2.0350000000000001,"5min":701.73400000000004,"15min":701.73400000000004},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.133,"5min":1.3400000000000001,"15min":1.26},"min":{"1min":0.75700000000000001,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.917,"5min":5.758,"15min":5.758},"last":0.76600000000000001}]},{"osd":6,"last update":"Fri Jan 29 13:56:29 2021","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.4419999999999999,"15min":1.3440000000000001},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.9139999999999999,"5min":5.7199999999999998,"15min":5.7199999999999998},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.258,"5min":4.0110000000000001,"15min":6.2149999999999999},"min":{"1min":0.72299999999999998,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.6230000000000002,"5min":132.35300000000001,"15min":132.35300000000001},"last":1.0589999999999999}]},{"osd":7,"last update":"Fri Jan 29 13:56:29 2021","interfaces":[{"interface":"back","average":{"1min":37.829999999999998,"5min":8.8330000000000002,"15min":3.823},"min":{"1min":0.81299999999999994,"5min":0.752,"15min":0.752},"max":{"1min":696.96400000000006,"5min":696.96400000000006,"15min":696.96400000000006},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.4990000000000001,"15min":1.413},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.0369999999999999,"5min":5.9450000000000003,"15min":5.9450000000000003},"last":0.85199999999999998}]}]},{"osd":3,"up_from":15,"seq":64424509501,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3443244,"kb_used_data":3391652,"kb_used_omap":0,"kb_used_meta":51584,"kb_avail":90928596,"statfs":{"total":96636764160,"available":93110882304,"internally_reserved":0,"allocated":3473051648,"data_stored":3742620723,"data_compressed":29735076,"data_compressed_allocated":290086912,"data_compressed_original":580173824,"omap_allocated":0,"internal_metadata":52822016},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:54:26 2021","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":0.93999999999999995,"15min":0.90400000000000003},"min":{"1min":0.67500000000000004,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.617,"5min":2.617,"15min":2.617},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.135,"5min":3.3719999999999999,"15min":3.7450000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":3.9620000000000002,"5min":60.329000000000001,"15min":60.329000000000001},"last":1.0529999999999999}]},{"osd":1,"last update":"Fri Jan 29 13:56:30 2021","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.97599999999999998,"15min":0.91300000000000003},"min":{"1min":0.497,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0109999999999999,"5min":2.5819999999999999,"15min":2.5819999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.82699999999999996,"5min":10.324,"15min":4.0970000000000004},"min":{"1min":0.63800000000000001,"5min":0.496,"15min":0.496},"max":{"1min":1.0629999999999999,"5min":977.78200000000004,"15min":977.78200000000004},"last":0.66900000000000004}]},{"osd":2,"last update":"Fri Jan 29 13:56:30 2021","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":1.208,"15min":1.0880000000000001},"min":{"1min":0.61099999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.22,"5min":5.7539999999999996,"15min":5.7539999999999996},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.018,"5min":1.383,"15min":1.4339999999999999},"min":{"1min":0.73199999999999998,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.4870000000000001,"5min":6.4509999999999996,"15min":6.4509999999999996},"last":0.96899999999999997}]},{"osd":4,"last update":"Fri Jan 29 13:56:11 2021","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":11.106,"15min":4.5620000000000003},"min":{"1min":0.70799999999999996,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.3979999999999999,"5min":916.09500000000003,"15min":916.09500000000003},"last":0.71099999999999997},{"interface":"front","average":{"1min":1.1479999999999999,"5min":11.696999999999999,"15min":4.8819999999999997},"min":{"1min":0.82099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.593,"5min":977.91499999999996,"15min":977.91499999999996},"last":0.92900000000000005}]},{"osd":5,"last update":"Fri Jan 29 13:56:30 2021","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":6.9089999999999998,"15min":3.1429999999999998},"min":{"1min":0.72799999999999998,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.8160000000000001,"5min":503.93000000000001,"15min":503.93000000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.077,"5min":6.79,"15min":3.0230000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.3799999999999999,"5min":503.74299999999999,"15min":503.74299999999999},"last":1.141}]},{"osd":6,"last update":"Fri Jan 29 13:56:30 2021","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.4490000000000001,"15min":1.351},"min":{"1min":0.874,"5min":0.751,"15min":0.751},"max":{"1min":3.2930000000000001,"5min":4.7539999999999996,"15min":4.7539999999999996},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.339,"5min":19.396999999999998,"15min":37.238999999999997},"min":{"1min":0.876,"5min":0.876,"15min":0.876},"max":{"1min":3.3690000000000002,"5min":851.89099999999996,"15min":851.89099999999996},"last":1.173}]},{"osd":7,"last update":"Fri Jan 29 13:56:30 2021","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.5609999999999999,"15min":1.5069999999999999},"min":{"1min":0.90800000000000003,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.613,"5min":6.4240000000000004,"15min":6.4240000000000004},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.8300000000000001,"15min":1.966},"min":{"1min":1.0629999999999999,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.635,"5min":15.597,"15min":15.597},"last":1.038}]}]},{"osd":4,"up_from":42,"seq":180388626482,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4411588,"kb_used_data":4363992,"kb_used_omap":0,"kb_used_meta":47423,"kb_avail":89960252,"statfs":{"total":96636764160,"available":92119298048,"internally_reserved":0,"allocated":4468727808,"data_stored":5613881037,"data_compressed":118370768,"data_compressed_allocated":1155686400,"data_compressed_original":2311372800,"omap_allocated":972,"internal_metadata":48561204},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:54:12 2021","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":4.2450000000000001,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.746},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":4.1589999999999998,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":1.141}]},{"osd":1,"last update":"Fri Jan 29 13:56:18 2021","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":43.862000000000002,"15min":62.893000000000001},"min":{"1min":0.76800000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":2.6850000000000001,"5min":924.63300000000004,"15min":924.63300000000004},"last":1.008},{"interface":"front","average":{"1min":74.046000000000006,"5min":15.648,"15min":5.9059999999999997},"min":{"1min":0.84299999999999997,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":898.86900000000003,"5min":898.86900000000003,"15min":898.86900000000003},"last":1.2090000000000001}]},{"osd":2,"last update":"Fri Jan 29 13:56:18 2021","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":11.952,"15min":4.7160000000000002},"min":{"1min":0.745,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.3149999999999999,"5min":857.78899999999999,"15min":857.78899999999999},"last":1.1679999999999999},{"interface":"front","average":{"1min":44.206000000000003,"5min":9.843,"15min":4.0510000000000002},"min":{"1min":0.68600000000000005,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":779.38400000000001,"5min":779.38400000000001,"15min":779.38400000000001},"last":0.78100000000000003}]},{"osd":3,"last update":"Fri Jan 29 13:56:18 2021","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":12.48,"15min":14.343},"min":{"1min":0.754,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":2.738,"5min":311.68299999999999,"15min":311.68299999999999},"last":1.079},{"interface":"front","average":{"1min":1.028,"5min":1.236,"15min":1.264},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.4359999999999999,"5min":3.6539999999999999,"15min":3.6539999999999999},"last":1.2509999999999999}]},{"osd":5,"last update":"Fri Jan 29 13:56:18 2021","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":3.7309999999999999,"15min":1.964},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.5109999999999999,"5min":239.99199999999999,"15min":239.99199999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.171,"15min":1.1599999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":3.1749999999999998,"5min":3.1749999999999998,"15min":3.1749999999999998},"last":1.03}]},{"osd":6,"last update":"Fri Jan 29 13:56:18 2021","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.238,"15min":1.206},"min":{"1min":0.72499999999999998,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.7789999999999999,"5min":5.5890000000000004,"15min":5.5890000000000004},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.252,"15min":1.296},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.657,"5min":3.6880000000000002,"15min":3.6880000000000002},"last":0.91600000000000004}]},{"osd":7,"last update":"Fri Jan 29 13:56:18 2021","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.3,"15min":1.286},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":2.988,"5min":3.0169999999999999,"15min":3.0169999999999999},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.3380000000000001,"15min":1.3400000000000001},"min":{"1min":0.96099999999999997,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":1.635,"5min":3.6699999999999999,"15min":3.6699999999999999},"last":1.357}]}]},{"osd":5,"up_from":15,"seq":64424509500,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":765812,"kb_used_data":751852,"kb_used_omap":0,"kb_used_meta":13952,"kb_avail":93606028,"statfs":{"total":96636764160,"available":95852572672,"internally_reserved":0,"allocated":769896448,"data_stored":1055801421,"data_compressed":29380114,"data_compressed_allocated":286253056,"data_compressed_original":572506112,"omap_allocated":0,"internal_metadata":14286848},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:54:20 2021","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.83999999999999997,"15min":0.81299999999999994},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":2.3250000000000002,"5min":2.3250000000000002,"15min":2.3250000000000002},"last":1.079},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.101,"15min":1.044},"min":{"1min":0.65600000000000003,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":4.8410000000000002,"5min":4.8410000000000002,"15min":4.8410000000000002},"last":1.306}]},{"osd":1,"last update":"Fri Jan 29 13:56:20 2021","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":0.93600000000000005,"15min":0.85999999999999999},"min":{"1min":0.68500000000000005,"5min":0.498,"15min":0.498},"max":{"1min":2.3690000000000002,"5min":3.1819999999999999,"15min":3.1819999999999999},"last":1.1140000000000001},{"interface":"front","average":{"1min":51.244999999999997,"5min":11.042999999999999,"15min":4.2709999999999999},"min":{"1min":0.82999999999999996,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":732.80700000000002,"5min":732.80700000000002,"15min":732.80700000000002},"last":1.365}]},{"osd":2,"last update":"Fri Jan 29 13:56:20 2021","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":8.1699999999999999,"15min":3.548},"min":{"1min":0.73599999999999999,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.161,"5min":649.82899999999995,"15min":649.82899999999995},"last":1.6950000000000001},{"interface":"front","average":{"1min":1.095,"5min":1.1180000000000001,"15min":1.054},"min":{"1min":0.70699999999999996,"5min":0.623,"15min":0.623},"max":{"1min":2.4169999999999998,"5min":4.1799999999999997,"15min":4.1799999999999997},"last":1.145}]},{"osd":3,"last update":"Fri Jan 29 13:56:20 2021","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":1.137,"15min":1.133},"min":{"1min":0.626,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.2829999999999999,"5min":4.0890000000000004,"15min":4.0890000000000004},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.274,"15min":1.23},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":2.1200000000000001,"5min":4.827,"15min":4.827},"last":1.7350000000000001}]},{"osd":4,"last update":"Fri Jan 29 13:56:17 2021","interfaces":[{"interface":"back","average":{"1min":28.036000000000001,"5min":16.495999999999999,"15min":17.274999999999999},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":514.74199999999996,"5min":514.74199999999996,"15min":514.74199999999996},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.2430000000000001,"15min":1.204},"min":{"1min":0.86699999999999999,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":2.266,"5min":4.4359999999999999,"15min":4.4359999999999999},"last":1.7689999999999999}]},{"osd":6,"last update":"Fri Jan 29 13:56:20 2021","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":9.5289999999999999,"15min":4.0229999999999997},"min":{"1min":0.81499999999999995,"5min":0.622,"15min":0.622},"max":{"1min":2.3039999999999998,"5min":520.71799999999996,"15min":520.71799999999996},"last":1.804},{"interface":"front","average":{"1min":1.345,"5min":1.421,"15min":1.256},"min":{"1min":0.84399999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.3929999999999998,"5min":4.8700000000000001,"15min":4.8700000000000001},"last":1.8320000000000001}]},{"osd":7,"last update":"Fri Jan 29 13:56:20 2021","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.4710000000000001,"15min":1.5209999999999999},"min":{"1min":0.92700000000000005,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.3420000000000001,"5min":9.0109999999999992,"15min":9.0109999999999992},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.383,"5min":1.7270000000000001,"15min":1.7849999999999999},"min":{"1min":1.0049999999999999,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":2.3319999999999999,"5min":9.593,"15min":9.593},"last":1.96}]}]},{"osd":6,"up_from":15,"seq":64424509500,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4180688,"kb_used_data":4117640,"kb_used_omap":0,"kb_used_meta":63040,"kb_avail":90191152,"statfs":{"total":96636764160,"available":92355739648,"internally_reserved":0,"allocated":4216463360,"data_stored":4766148518,"data_compressed":58393750,"data_compressed_allocated":570441728,"data_compressed_original":1140883456,"omap_allocated":0,"internal_metadata":64552960},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:54:20 2021","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.379,"15min":1.381},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":3.0899999999999999,"5min":5.8380000000000001,"15min":5.8380000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":1.286,"5min":1.012,"15min":0.96599999999999997},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":7.3810000000000002,"5min":7.3810000000000002,"15min":7.3810000000000002},"last":0.50800000000000001}]},{"osd":1,"last update":"Fri Jan 29 13:56:22 2021","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":15.146000000000001,"15min":29.238},"min":{"1min":0.65400000000000003,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.353,"5min":684.42700000000002,"15min":684.42700000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.2849999999999999,"15min":1.1699999999999999},"min":{"1min":0.58899999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.306,"5min":6.1360000000000001,"15min":6.1360000000000001},"last":0.755}]},{"osd":2,"last update":"Fri Jan 29 13:56:22 2021","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.1919999999999999,"15min":1.242},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.492,"5min":7.2329999999999997,"15min":7.2329999999999997},"last":1.133},{"interface":"front","average":{"1min":36.773000000000003,"5min":8.3480000000000008,"15min":3.6339999999999999},"min":{"1min":0.68799999999999994,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":607.71000000000004,"5min":607.71000000000004,"15min":607.71000000000004},"last":0.67700000000000005}]},{"osd":3,"last update":"Fri Jan 29 13:56:22 2021","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":1.345,"15min":1.2470000000000001},"min":{"1min":0.66800000000000004,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.1779999999999999,"5min":6.048,"15min":6.048},"last":200.50399999999999},{"interface":"front","average":{"1min":29.335999999999999,"5min":6.8609999999999998,"15min":3.0449999999999999},"min":{"1min":0.83699999999999997,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":480.87299999999999,"5min":480.87299999999999,"15min":480.87299999999999},"last":1.21}]},{"osd":4,"last update":"Fri Jan 29 13:56:12 2021","interfaces":[{"interface":"back","average":{"1min":17.359999999999999,"5min":4.6619999999999999,"15min":2.456},"min":{"1min":0.629,"5min":0.62,"15min":0.62},"max":{"1min":263.584,"5min":263.584,"15min":263.584},"last":1.173},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.3069999999999999,"15min":1.417},"min":{"1min":0.78900000000000003,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.474,"5min":7.3449999999999998,"15min":7.3449999999999998},"last":0.82699999999999996}]},{"osd":5,"last update":"Fri Jan 29 13:56:22 2021","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.2969999999999999,"15min":1.349},"min":{"1min":0.86499999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.496,"5min":4.0069999999999997,"15min":4.0069999999999997},"last":1.034},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.6100000000000001,"15min":1.506},"min":{"1min":0.96499999999999997,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":1.3759999999999999,"5min":7.3140000000000001,"15min":7.3140000000000001},"last":1.052}]},{"osd":7,"last update":"Fri Jan 29 13:56:22 2021","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.4159999999999999,"15min":1.3380000000000001},"min":{"1min":0.90100000000000002,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.655,"5min":10.795,"15min":10.795},"last":1.375},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.3879999999999999,"15min":1.3919999999999999},"min":{"1min":0.85899999999999999,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.609,"5min":4.0579999999999998,"15min":4.0579999999999998},"last":0.99299999999999999}]}]}]}} , 2021-01-29T13:57:11.433 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-29T13:57:11.716 INFO:teuthology.orchestra.run.smithi183.stdout: 2021-01-30T06:14:57.497 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T13:57:44.175 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":301,"stamp":"2021-01-29T13:57:42.675417+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20305121604,"num_objects":6822,"num_object_clones":0,"num_object_copies":19368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":632,"num_objects_unfound":0,"num_objects_dirty":4824,"num_whiteouts":0,"num_read":9651,"num_read_kb":4830,"num_write":25509,"num_write_kb":14727220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3207,"num_bytes_recovered":7385874976,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":21,"num_bytes_hit_set_archive":12196,"num_flush":2862,"num_flush_kb":8348668,"num_evict":884,"num_evict_kb":2571590,"num_promote":4835,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8249,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15129,"ondisk_log_size":15129,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":45,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":25551644,"kb_used_data":25207644,"kb_used_omap":1,"kb_used_meta":343934,"kb_avail":729423076,"statfs":{"total":773094113280,"available":746929229824,"internally_reserved":0,"allocated":25812627456,"data_stored":30826570098,"data_compressed":523387886,"data_compressed_allocated":5107855360,"data_compressed_original":10215710720,"omap_allocated":1944,"internal_metadata":352188520},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-252,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":339084355,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.225468"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":"156","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.458800+0000","last_change":"2021-01-29T13:57:34.166076+0000","last_active":"2021-01-29T13:57:37.458800+0000","last_peered":"2021-01-29T13:57:37.458800+0000","last_clean":"2021-01-29T13:57:37.458800+0000","last_became_active":"2021-01-29T13:52:24.188931+0000","last_became_peered":"2021-01-29T13:52:24.188931+0000","last_unstale":"2021-01-29T13:57:37.458800+0000","last_undegraded":"2021-01-29T13:57:37.458800+0000","last_fullsized":"2021-01-29T13:57:37.458800+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:55:57.162808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:55:57.162808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"120'6449","reported_seq":"8600","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.461338+0000","last_change":"2021-01-29T13:55:45.625490+0000","last_active":"2021-01-29T13:57:37.461338+0000","last_peered":"2021-01-29T13:57:37.461338+0000","last_clean":"2021-01-29T13:57:37.461338+0000","last_became_active":"2021-01-29T13:52:39.830011+0000","last_became_peered":"2021-01-29T13:52:39.830011+0000","last_unstale":"2021-01-29T13:57:37.461338+0000","last_undegraded":"2021-01-29T13:57:37.461338+0000","last_fullsized":"2021-01-29T13:57:37.461338+0000","mapping_epoch":26,"log_start":"72'3403","ondisk_log_start":"72'3403","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'6250","last_scrub_stamp":"2021-01-29T13:55:45.625379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:38.811171+0000","last_clean_scrub_stamp":"2021-01-29T13:55:45.625379+0000","log_size":3046,"ondisk_log_size":3046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2928382292,"num_objects":986,"num_object_clones":0,"num_object_copies":1972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":2501,"num_read_kb":1252,"num_write":5879,"num_write_kb":1639641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":3315,"num_flush":761,"num_flush_kb":2192275,"num_evict":269,"num_evict_kb":760560,"num_promote":1249,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"117","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.497927+0000","last_change":"2021-01-29T13:55:42.661441+0000","last_active":"2021-01-29T13:57:37.497927+0000","last_peered":"2021-01-29T13:57:37.497927+0000","last_clean":"2021-01-29T13:57:37.497927+0000","last_became_active":"2021-01-29T13:53:29.493255+0000","last_became_peered":"2021-01-29T13:53:29.493255+0000","last_unstale":"2021-01-29T13:57:37.497927+0000","last_undegraded":"2021-01-29T13:57:37.497927+0000","last_fullsized":"2021-01-29T13:57:37.497927+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:55:42.661371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:55:42.661371+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3","version":"120'671","reported_seq":"1393","reported_epoch":"147","state":"active+remapped+backfilling","last_fresh":"2021-01-29T13:57:40.662249+0000","last_change":"2021-01-29T13:56:47.680232+0000","last_active":"2021-01-29T13:57:40.662249+0000","last_peered":"2021-01-29T13:57:40.662249+0000","last_clean":"2021-01-29T13:56:36.386610+0000","last_became_active":"2021-01-29T13:56:47.674484+0000","last_became_peered":"2021-01-29T13:56:47.674484+0000","last_unstale":"2021-01-29T13:57:40.662249+0000","last_undegraded":"2021-01-29T13:57:40.662249+0000","last_fullsized":"2021-01-29T13:57:40.662249+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'570","last_scrub_stamp":"2021-01-29T13:55:44.911108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:35.792587+0000","last_clean_scrub_stamp":"2021-01-29T13:55:44.911108+0000","log_size":671,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2038945493,"num_objects":671,"num_object_clones":0,"num_object_copies":2684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":404,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":671,"num_write_kb":1991482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":787,"num_bytes_recovered":2413500764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4,7],"acting":[6,2,4,7],"avail_no_missing":["6(0)","2(1)","4(2)","7(3)"],"object_location_counts":[{"shards":"2(1),4(2),6(0),7(3)","objects":671}],"blocked_by":[],"up_primary":0,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"147","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.457918+0000","last_change":"2021-01-29T13:57:01.550191+0000","last_active":"2021-01-29T13:57:37.457918+0000","last_peered":"2021-01-29T13:57:37.457918+0000","last_clean":"2021-01-29T13:57:37.457918+0000","last_became_active":"2021-01-29T13:52:23.765595+0000","last_became_peered":"2021-01-29T13:52:23.765595+0000","last_unstale":"2021-01-29T13:57:37.457918+0000","last_undegraded":"2021-01-29T13:57:37.457918+0000","last_fullsized":"2021-01-29T13:57:37.457918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:57:01.550087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:57:01.550087+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"117'710","reported_seq":"887","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.458693+0000","last_change":"2021-01-29T13:57:37.153012+0000","last_active":"2021-01-29T13:57:37.458693+0000","last_peered":"2021-01-29T13:57:37.458693+0000","last_clean":"2021-01-29T13:57:37.458693+0000","last_became_active":"2021-01-29T13:53:09.969063+0000","last_became_peered":"2021-01-29T13:53:09.969063+0000","last_unstale":"2021-01-29T13:57:37.458693+0000","last_undegraded":"2021-01-29T13:57:37.458693+0000","last_fullsized":"2021-01-29T13:57:37.458693+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'710","last_scrub_stamp":"2021-01-29T13:56:09.327903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:35.792587+0000","last_clean_scrub_stamp":"2021-01-29T13:56:09.327903+0000","log_size":710,"ondisk_log_size":710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2107977089,"num_objects":710,"num_object_clones":0,"num_object_copies":2840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":710,"num_write_kb":2058922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2,4],"acting":[5,7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"159","reported_epoch":"145","state":"active+clean","last_fresh":"2021-01-29T13:57:36.794315+0000","last_change":"2021-01-29T13:56:59.467725+0000","last_active":"2021-01-29T13:57:36.794315+0000","last_peered":"2021-01-29T13:57:36.794315+0000","last_clean":"2021-01-29T13:57:36.794315+0000","last_became_active":"2021-01-29T13:52:24.188578+0000","last_became_peered":"2021-01-29T13:52:24.188578+0000","last_unstale":"2021-01-29T13:57:36.794315+0000","last_undegraded":"2021-01-29T13:57:36.794315+0000","last_fullsized":"2021-01-29T13:57:36.794315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:56:59.467654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:56:59.467654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"142'4","reported_seq":"157","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.458197+0000","last_change":"2021-01-29T13:57:01.982145+0000","last_active":"2021-01-29T13:57:37.458197+0000","last_peered":"2021-01-29T13:57:37.458197+0000","last_clean":"2021-01-29T13:57:37.458197+0000","last_became_active":"2021-01-29T13:52:20.655015+0000","last_became_peered":"2021-01-29T13:52:20.655015+0000","last_unstale":"2021-01-29T13:57:37.458197+0000","last_undegraded":"2021-01-29T13:57:37.458197+0000","last_fullsized":"2021-01-29T13:57:37.458197+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'4","last_scrub_stamp":"2021-01-29T13:57:01.982073+0000","last_deep_scrub":"142'4","last_deep_scrub_stamp":"2021-01-29T13:57:01.982073+0000","last_clean_scrub_stamp":"2021-01-29T13:57:01.982073+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8249,"num_omap_keys":2,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"120'761","reported_seq":"3071","reported_epoch":"147","state":"active+remapped+backfilling","last_fresh":"2021-01-29T13:57:41.858978+0000","last_change":"2021-01-29T13:56:48.363777+0000","last_active":"2021-01-29T13:57:41.858978+0000","last_peered":"2021-01-29T13:57:41.858978+0000","last_clean":"2021-01-29T13:56:36.386422+0000","last_became_active":"2021-01-29T13:56:48.362095+0000","last_became_peered":"2021-01-29T13:56:48.362095+0000","last_unstale":"2021-01-29T13:57:41.858978+0000","last_undegraded":"2021-01-29T13:57:41.858978+0000","last_fullsized":"2021-01-29T13:57:41.858978+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'761","last_scrub_stamp":"2021-01-29T13:55:53.981049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:35.792587+0000","last_clean_scrub_stamp":"2021-01-29T13:55:53.981049+0000","log_size":761,"ondisk_log_size":761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2244508217,"num_objects":761,"num_object_clones":0,"num_object_copies":3044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":228,"num_objects_unfound":0,"num_objects_dirty":761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":761,"num_write_kb":2192275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1053,"num_bytes_recovered":3106526858,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7,0],"acting":[4,2,7,1],"avail_no_missing":["4(0)","1(3)","2(1)","7(2)"],"object_location_counts":[{"shards":"1(3),2(1),4(0),7(2)","objects":761}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"148","reported_epoch":"145","state":"active+clean","last_fresh":"2021-01-29T13:57:36.794261+0000","last_change":"2021-01-29T13:56:41.371116+0000","last_active":"2021-01-29T13:57:36.794261+0000","last_peered":"2021-01-29T13:57:36.794261+0000","last_clean":"2021-01-29T13:57:36.794261+0000","last_became_active":"2021-01-29T13:52:23.765671+0000","last_became_peered":"2021-01-29T13:52:23.765671+0000","last_unstale":"2021-01-29T13:57:36.794261+0000","last_undegraded":"2021-01-29T13:57:36.794261+0000","last_fullsized":"2021-01-29T13:57:36.794261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:56:41.371025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:56:41.371025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"118'720","reported_seq":"891","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.461342+0000","last_change":"2021-01-29T13:55:59.202455+0000","last_active":"2021-01-29T13:57:37.461342+0000","last_peered":"2021-01-29T13:57:37.461342+0000","last_clean":"2021-01-29T13:57:37.461342+0000","last_became_active":"2021-01-29T13:53:09.587252+0000","last_became_peered":"2021-01-29T13:53:09.587252+0000","last_unstale":"2021-01-29T13:57:37.461342+0000","last_undegraded":"2021-01-29T13:57:37.461342+0000","last_fullsized":"2021-01-29T13:57:37.461342+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'720","last_scrub_stamp":"2021-01-29T13:55:59.202349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:35.792587+0000","last_clean_scrub_stamp":"2021-01-29T13:55:59.202349+0000","log_size":720,"ondisk_log_size":720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2156154079,"num_objects":720,"num_object_clones":0,"num_object_copies":2880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":720,"num_write_kb":2105989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4,3],"acting":[6,2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"120'5986","reported_seq":"10744","reported_epoch":"147","state":"active+clean","last_fresh":"2021-01-29T13:57:38.478514+0000","last_change":"2021-01-29T13:57:38.478514+0000","last_active":"2021-01-29T13:57:38.478514+0000","last_peered":"2021-01-29T13:57:38.478514+0000","last_clean":"2021-01-29T13:57:38.478514+0000","last_became_active":"2021-01-29T13:57:38.475245+0000","last_became_peered":"2021-01-29T13:57:38.475245+0000","last_unstale":"2021-01-29T13:57:38.478514+0000","last_undegraded":"2021-01-29T13:57:38.478514+0000","last_fullsized":"2021-01-29T13:57:38.478514+0000","mapping_epoch":146,"log_start":"71'2904","ondisk_log_start":"71'2904","created":26,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'5955","last_scrub_stamp":"2021-01-29T13:55:51.275882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:38.811171+0000","last_clean_scrub_stamp":"2021-01-29T13:55:51.275882+0000","log_size":3082,"ondisk_log_size":3082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2947693900,"num_objects":985,"num_object_clones":0,"num_object_copies":1970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":2316,"num_read_kb":1158,"num_write":5435,"num_write_kb":1544566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1367,"num_bytes_recovered":1865847354,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":2771,"num_flush":671,"num_flush_kb":1991482,"num_evict":178,"num_evict_kb":526959,"num_promote":1172,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"171","reported_epoch":"147","state":"active+clean","last_fresh":"2021-01-29T13:57:39.387290+0000","last_change":"2021-01-29T13:57:39.387896+0000","last_active":"2021-01-29T13:57:39.387290+0000","last_peered":"2021-01-29T13:57:39.387290+0000","last_clean":"2021-01-29T13:57:39.387290+0000","last_became_active":"2021-01-29T13:53:09.377717+0000","last_became_peered":"2021-01-29T13:53:09.377717+0000","last_unstale":"2021-01-29T13:57:39.387290+0000","last_undegraded":"2021-01-29T13:57:39.387290+0000","last_fullsized":"2021-01-29T13:57:39.387290+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:55:33.131622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:55:33.131622+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"145","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.458896+0000","last_change":"2021-01-29T13:56:53.807344+0000","last_active":"2021-01-29T13:57:37.458896+0000","last_peered":"2021-01-29T13:57:37.458896+0000","last_clean":"2021-01-29T13:57:37.458896+0000","last_became_active":"2021-01-29T13:52:40.828334+0000","last_became_peered":"2021-01-29T13:52:40.828334+0000","last_unstale":"2021-01-29T13:57:37.458896+0000","last_undegraded":"2021-01-29T13:57:37.458896+0000","last_fullsized":"2021-01-29T13:57:37.458896+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:56:53.807251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:56:53.807251+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"117'6273","reported_seq":"8324","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.497913+0000","last_change":"2021-01-29T13:55:55.445357+0000","last_active":"2021-01-29T13:57:37.497913+0000","last_peered":"2021-01-29T13:57:37.497913+0000","last_clean":"2021-01-29T13:57:37.497913+0000","last_became_active":"2021-01-29T13:53:09.369472+0000","last_became_peered":"2021-01-29T13:53:09.369472+0000","last_unstale":"2021-01-29T13:57:37.497913+0000","last_undegraded":"2021-01-29T13:57:37.497913+0000","last_fullsized":"2021-01-29T13:57:37.497913+0000","mapping_epoch":42,"log_start":"71'3203","ondisk_log_start":"71'3203","created":26,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'6273","last_scrub_stamp":"2021-01-29T13:55:55.445229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:38.811171+0000","last_clean_scrub_stamp":"2021-01-29T13:55:55.445229+0000","log_size":3070,"ondisk_log_size":3070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2935339657,"num_objects":992,"num_object_clones":0,"num_object_copies":1984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":496,"num_whiteouts":0,"num_read":2414,"num_read_kb":1208,"num_write":5647,"num_write_kb":1593121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":3044,"num_flush":710,"num_flush_kb":2058922,"num_evict":219,"num_evict_kb":640347,"num_promote":1206,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"29","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.463934+0000","last_change":"2021-01-29T13:56:46.442031+0000","last_active":"2021-01-29T13:57:37.463934+0000","last_peered":"2021-01-29T13:57:37.463934+0000","last_clean":"2021-01-29T13:57:37.463934+0000","last_became_active":"2021-01-29T13:56:46.441589+0000","last_became_peered":"2021-01-29T13:56:46.441589+0000","last_unstale":"2021-01-29T13:57:37.463934+0000","last_undegraded":"2021-01-29T13:57:37.463934+0000","last_fullsized":"2021-01-29T13:57:37.463934+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:55:55.134762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:55:55.134762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"118'6270","reported_seq":"8356","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.464735+0000","last_change":"2021-01-29T13:57:04.235378+0000","last_active":"2021-01-29T13:57:37.464735+0000","last_peered":"2021-01-29T13:57:37.464735+0000","last_clean":"2021-01-29T13:57:37.464735+0000","last_became_active":"2021-01-29T13:52:39.823642+0000","last_became_peered":"2021-01-29T13:52:39.823642+0000","last_unstale":"2021-01-29T13:57:37.464735+0000","last_undegraded":"2021-01-29T13:57:37.464735+0000","last_fullsized":"2021-01-29T13:57:37.464735+0000","mapping_epoch":26,"log_start":"71'3205","ondisk_log_start":"71'3205","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6270","last_scrub_stamp":"2021-01-29T13:57:04.235268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:38.811171+0000","last_clean_scrub_stamp":"2021-01-29T13:57:04.235268+0000","log_size":3065,"ondisk_log_size":3065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2946120877,"num_objects":995,"num_object_clones":0,"num_object_copies":1990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":2420,"num_read_kb":1212,"num_write":5684,"num_write_kb":1601214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":3066,"num_flush":720,"num_flush_kb":2105989,"num_evict":218,"num_evict_kb":643724,"num_promote":1208,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":4,"stat_sum":{"num_bytes":11757536726,"num_objects":3958,"num_object_clones":0,"num_object_copies":7916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1960,"num_whiteouts":0,"num_read":9651,"num_read_kb":4830,"num_write":22645,"num_write_kb":6378542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1367,"num_bytes_recovered":1865847354,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":21,"num_bytes_hit_set_archive":12196,"num_flush":2862,"num_flush_kb":8348668,"num_evict":884,"num_evict_kb":2571590,"num_promote":4835,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11983056896,"data_stored":11892361364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12263,"ondisk_log_size":12263,"up":8,"acting":8,"num_store_stats":6},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":8547584878,"num_objects":2862,"num_object_clones":0,"num_object_copies":11448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":632,"num_objects_unfound":0,"num_objects_dirty":2862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2862,"num_write_kb":8348668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1840,"num_bytes_recovered":5520027622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13793206272,"data_stored":18905419776,"data_compressed":523841489,"data_compressed_allocated":5112213504,"data_compressed_original":10224427008,"omap_allocated":0,"internal_metadata":0},"log_size":2862,"ondisk_log_size":2862,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8249,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":0,"up_from":124,"seq":532575944719,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2622556,"kb_used_data":2579220,"kb_used_omap":0,"kb_used_meta":43327,"kb_avail":91749284,"statfs":{"total":96636764160,"available":93951266816,"internally_reserved":0,"allocated":2641121280,"data_stored":3110022318,"data_compressed":49186610,"data_compressed_allocated":479526912,"data_compressed_original":959053824,"omap_allocated":972,"internal_metadata":44366900},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":1.0149999999999999}]},{"osd":2,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":2.3559999999999999,"5min":2.3559999999999999,"15min":2.3559999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.93300000000000005}]},{"osd":3,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":7.8730000000000002,"15min":7.8730000000000002},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":170.62200000000001,"5min":170.62200000000001,"15min":170.62200000000001},"last":170.62200000000001},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":2.3149999999999999,"5min":2.3149999999999999,"15min":2.3149999999999999},"last":1.258}]},{"osd":4,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.117,"15min":1.117},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":1.056},{"interface":"front","average":{"1min":1.141,"5min":1.141,"15min":1.141},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.1059999999999999,"5min":2.1059999999999999,"15min":2.1059999999999999},"last":1.2370000000000001}]},{"osd":5,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":6.9889999999999999,"5min":6.9889999999999999,"15min":6.9889999999999999},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":136.00800000000001,"5min":136.00800000000001,"15min":136.00800000000001},"last":1.294},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":0.96299999999999997}]},{"osd":6,"last update":"Fri Jan 29 13:57:37 2021","interfaces":[{"interface":"back","average":{"1min":22.164000000000001,"5min":22.164000000000001,"15min":22.164000000000001},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":484.286,"5min":484.286,"15min":484.286},"last":1.1140000000000001},{"interface":"front","average":{"1min":24.145,"5min":24.145,"15min":24.145},"min":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"max":{"1min":529.46000000000004,"5min":529.46000000000004,"15min":529.46000000000004},"last":529.46000000000004}]},{"osd":7,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.274,"15min":1.274},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"max":{"1min":2.3879999999999999,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.264,"5min":1.264,"15min":1.264},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.349}]}]},{"osd":7,"up_from":15,"seq":64424509508,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2311472,"kb_used_data":2281064,"kb_used_omap":0,"kb_used_meta":30400,"kb_avail":92060368,"statfs":{"total":96636764160,"available":94269816832,"internally_reserved":0,"allocated":2335809536,"data_stored":3200494326,"data_compressed":88587048,"data_compressed_allocated":865042432,"data_compressed_original":1730084864,"omap_allocated":0,"internal_metadata":31129600},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.70299999999999996},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":1.004}]},{"osd":1,"last update":"Fri Jan 29 13:57:30 2021","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.242,"15min":1.46},"min":{"1min":0.64900000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.671,"5min":11.195,"15min":11.195},"last":0.91200000000000003},{"interface":"front","average":{"1min":50.901000000000003,"5min":26.797999999999998,"15min":33.463000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":951.06700000000001,"5min":951.06700000000001,"15min":951.06700000000001},"last":0.79000000000000004}]},{"osd":2,"last update":"Fri Jan 29 13:57:30 2021","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":10.314,"15min":4.5030000000000001},"min":{"1min":0.72099999999999997,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.147,"5min":512.423,"15min":512.423},"last":0.73799999999999999},{"interface":"front","average":{"1min":27.712,"5min":7.3419999999999996,"15min":3.5150000000000001},"min":{"1min":0.68600000000000005,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":505.935,"5min":505.935,"15min":505.935},"last":0.95299999999999996}]},{"osd":3,"last update":"Fri Jan 29 13:57:30 2021","interfaces":[{"interface":"back","average":{"1min":52.369999999999997,"5min":13.276,"15min":5.1639999999999997},"min":{"1min":0.749,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":977.31200000000001,"5min":977.31200000000001,"15min":977.31200000000001},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.381,"15min":1.581},"min":{"1min":1.006,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.7789999999999999,"5min":11.146000000000001,"15min":11.146000000000001},"last":1.3049999999999999}]},{"osd":4,"last update":"Fri Jan 29 13:57:17 2021","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.254,"15min":1.3340000000000001},"min":{"1min":0.79900000000000004,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.7450000000000001,"5min":5.8449999999999998,"15min":5.8449999999999998},"last":1.0469999999999999},{"interface":"front","average":{"1min":16.503,"5min":13.058,"15min":19.832000000000001},"min":{"1min":0.79700000000000004,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":309.072,"5min":391.66699999999997,"15min":391.66699999999997},"last":0.98199999999999998}]},{"osd":5,"last update":"Fri Jan 29 13:57:30 2021","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":3.3260000000000001,"15min":2.242},"min":{"1min":0.76700000000000002,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.625,"5min":175.83199999999999,"15min":175.83199999999999},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.294,"15min":1.2130000000000001},"min":{"1min":0.97099999999999997,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.8120000000000001,"5min":4.4020000000000001,"15min":4.4020000000000001},"last":1.3260000000000001}]},{"osd":6,"last update":"Fri Jan 29 13:57:30 2021","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.552,"15min":1.532},"min":{"1min":1.119,"5min":0.754,"15min":0.754},"max":{"1min":1.881,"5min":10.063000000000001,"15min":10.063000000000001},"last":1.345},{"interface":"front","average":{"1min":8.1750000000000007,"5min":9.5399999999999991,"15min":26.190000000000001},"min":{"1min":0.95299999999999996,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":132.83500000000001,"5min":623.01999999999998,"15min":623.01999999999998},"last":1.278}]}]},{"osd":2,"up_from":15,"seq":64424509507,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5660992,"kb_used_data":5606840,"kb_used_omap":0,"kb_used_meta":54144,"kb_avail":88710848,"statfs":{"total":96636764160,"available":90839908352,"internally_reserved":0,"allocated":5741404160,"data_stored":6879489840,"data_compressed":118712623,"data_compressed_allocated":1158184960,"data_compressed_original":2316369920,"omap_allocated":0,"internal_metadata":55443456},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.94299999999999995}]},{"osd":1,"last update":"Fri Jan 29 13:57:29 2021","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":17.460000000000001,"15min":6.4560000000000004},"min":{"1min":0.63200000000000001,"5min":0.499,"15min":0.499},"max":{"1min":1.3999999999999999,"5min":721.11000000000001,"15min":721.11000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.95499999999999996,"5min":21.443999999999999,"15min":7.8150000000000004},"min":{"1min":0.71899999999999997,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.3460000000000001,"5min":1000.376,"15min":1000.376},"last":0.96999999999999997}]},{"osd":3,"last update":"Fri Jan 29 13:57:29 2021","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":11.378,"15min":4.343},"min":{"1min":0.60299999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1160000000000001,"5min":501.61200000000002,"15min":501.61200000000002},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.96799999999999997,"5min":1.1819999999999999,"15min":1.0620000000000001},"min":{"1min":0.65900000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.3500000000000001,"5min":6.1870000000000003,"15min":6.1870000000000003},"last":0.90100000000000002}]},{"osd":4,"last update":"Fri Jan 29 13:57:18 2021","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.4099999999999999,"15min":1.5069999999999999},"min":{"1min":0.72699999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.6160000000000001,"5min":3.694,"15min":3.694},"last":1.1850000000000001},{"interface":"front","average":{"1min":77.519000000000005,"5min":16.635999999999999,"15min":6.4619999999999997},"min":{"1min":0.96199999999999997,"5min":0.871,"15min":0.871},"max":{"1min":874.88400000000001,"5min":874.88400000000001,"15min":874.88400000000001},"last":1.2050000000000001}]},{"osd":5,"last update":"Fri Jan 29 13:57:29 2021","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":5.7699999999999996,"15min":2.762},"min":{"1min":0.79800000000000004,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.349,"5min":183.761,"15min":183.761},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.1579999999999999,"5min":9.484,"15min":4.0389999999999997},"min":{"1min":0.88400000000000001,"5min":0.746,"15min":0.746},"max":{"1min":1.5469999999999999,"5min":521.02499999999998,"15min":521.02499999999998},"last":0.99099999999999999}]},{"osd":6,"last update":"Fri Jan 29 13:57:29 2021","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.359,"15min":1.2969999999999999},"min":{"1min":0.93500000000000005,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.544,"5min":3.46,"15min":3.46},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.135,"5min":1.4319999999999999,"15min":1.276},"min":{"1min":0.87,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.4590000000000001,"5min":7.0209999999999999,"15min":7.0209999999999999},"last":0.88500000000000001}]},{"osd":7,"last update":"Fri Jan 29 13:57:29 2021","interfaces":[{"interface":"back","average":{"1min":22.652000000000001,"5min":12.759,"15min":5.1719999999999997},"min":{"1min":0.98399999999999999,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":427.892,"5min":591.18899999999996,"15min":591.18899999999996},"last":1.226},{"interface":"front","average":{"1min":1.26,"5min":11.763999999999999,"15min":4.8159999999999998},"min":{"1min":0.95499999999999996,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.738,"5min":923.40099999999995,"15min":923.40099999999995},"last":1.103}]}]},{"osd":1,"up_from":15,"seq":64424509508,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2154968,"kb_used_data":2115152,"kb_used_omap":0,"kb_used_meta":39808,"kb_avail":92216872,"statfs":{"total":96636764160,"available":94430076928,"internally_reserved":0,"allocated":2165915648,"data_stored":2457997850,"data_compressed":31021897,"data_compressed_allocated":302632960,"data_compressed_original":605265920,"omap_allocated":0,"internal_metadata":40763392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:54:25 2021","interfaces":[{"interface":"back","average":{"1min":1.7629999999999999,"5min":1.0349999999999999,"15min":0.91300000000000003},"min":{"1min":0.63300000000000001,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":5.0629999999999997,"5min":5.0629999999999997,"15min":5.0629999999999997},"last":0.70799999999999996},{"interface":"front","average":{"1min":39.090000000000003,"5min":8.5060000000000002,"15min":3.4079999999999999},"min":{"1min":0.58699999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":752.73400000000004,"5min":752.73400000000004,"15min":752.73400000000004},"last":0.65200000000000002}]},{"osd":2,"last update":"Fri Jan 29 13:57:29 2021","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":16.138999999999999,"15min":27.606000000000002},"min":{"1min":0.57199999999999995,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.153,"5min":832.21600000000001,"15min":832.21600000000001},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.85299999999999998,"5min":11.237,"15min":4.4100000000000001},"min":{"1min":0.59099999999999997,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.1870000000000001,"5min":899.43399999999997,"15min":899.43399999999997},"last":0.84199999999999997}]},{"osd":3,"last update":"Fri Jan 29 13:57:29 2021","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":1.296,"15min":1.0960000000000001},"min":{"1min":0.63300000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.3169999999999999,"5min":6.0579999999999998,"15min":6.0579999999999998},"last":0.876},{"interface":"front","average":{"1min":0.84299999999999997,"5min":3.3029999999999999,"15min":9.1440000000000001},"min":{"1min":0.55100000000000005,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.1220000000000001,"5min":244.60900000000001,"15min":244.60900000000001},"last":0.82499999999999996}]},{"osd":4,"last update":"Fri Jan 29 13:57:18 2021","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.325,"15min":1.5880000000000001},"min":{"1min":0.71399999999999997,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.254,"5min":5.242,"15min":5.242},"last":0.95099999999999996},{"interface":"front","average":{"1min":1.079,"5min":1.6839999999999999,"15min":2.0870000000000002},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.2310000000000001,"5min":6.2590000000000003,"15min":6.2590000000000003},"last":1.165}]},{"osd":5,"last update":"Fri Jan 29 13:57:29 2021","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":14.631,"15min":5.681},"min":{"1min":0.68300000000000005,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.3440000000000001,"5min":701.73400000000004,"15min":701.73400000000004},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.302,"15min":1.248},"min":{"1min":0.72899999999999998,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.359,"5min":5.758,"15min":5.758},"last":1.0940000000000001}]},{"osd":6,"last update":"Fri Jan 29 13:57:30 2021","interfaces":[{"interface":"back","average":{"1min":48.951999999999998,"5min":10.974,"15min":4.5209999999999999},"min":{"1min":0.89800000000000002,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":862.01999999999998,"5min":862.01999999999998,"15min":862.01999999999998},"last":1.046},{"interface":"front","average":{"1min":1.1659999999999999,"5min":2.7810000000000001,"15min":5.8049999999999997},"min":{"1min":0.83799999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.4790000000000001,"5min":132.35300000000001,"15min":132.35300000000001},"last":1.1459999999999999}]},{"osd":7,"last update":"Fri Jan 29 13:57:29 2021","interfaces":[{"interface":"back","average":{"1min":45.155999999999999,"5min":17.600000000000001,"15min":6.7450000000000001},"min":{"1min":0.83999999999999997,"5min":0.752,"15min":0.752},"max":{"1min":749.81700000000001,"5min":749.81700000000001,"15min":749.81700000000001},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.4530000000000001,"15min":1.3979999999999999},"min":{"1min":0.85199999999999998,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":1.4790000000000001,"5min":5.9450000000000003,"15min":5.9450000000000003},"last":1.216}]}]},{"osd":3,"up_from":15,"seq":64424509508,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3443368,"kb_used_data":3391712,"kb_used_omap":0,"kb_used_meta":51648,"kb_avail":90928472,"statfs":{"total":96636764160,"available":93110755328,"internally_reserved":0,"allocated":3473113088,"data_stored":3742649427,"data_compressed":29735076,"data_compressed_allocated":290086912,"data_compressed_original":580173824,"omap_allocated":0,"internal_metadata":52887552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":29.826000000000001,"5min":29.826000000000001,"15min":29.826000000000001},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":301.69099999999997,"5min":301.69099999999997,"15min":301.69099999999997},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.613,"5min":1.613,"15min":1.613},"last":0.66800000000000004}]},{"osd":1,"last update":"Fri Jan 29 13:57:31 2021","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.96599999999999997,"15min":0.91000000000000003},"min":{"1min":0.57699999999999996,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.1339999999999999,"5min":2.5819999999999999,"15min":2.5819999999999999},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.86499999999999999,"5min":10.300000000000001,"15min":4.0890000000000004},"min":{"1min":0.59899999999999998,"5min":0.496,"15min":0.496},"max":{"1min":1.0600000000000001,"5min":977.78200000000004,"15min":977.78200000000004},"last":0.88500000000000001}]},{"osd":2,"last update":"Fri Jan 29 13:57:31 2021","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.181,"15min":1.079},"min":{"1min":0.65700000000000003,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.3919999999999999,"5min":5.7539999999999996,"15min":5.7539999999999996},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.3009999999999999,"15min":1.407},"min":{"1min":0.82999999999999996,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.2190000000000001,"5min":6.4509999999999996,"15min":6.4509999999999996},"last":1.1819999999999999}]},{"osd":4,"last update":"Fri Jan 29 13:57:15 2021","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":11.065,"15min":4.5490000000000004},"min":{"1min":0.71099999999999997,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.4890000000000001,"5min":916.09500000000003,"15min":916.09500000000003},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.131,"5min":11.628,"15min":4.859},"min":{"1min":0.81899999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.46,"5min":977.91499999999996,"15min":977.91499999999996},"last":1.0169999999999999}]},{"osd":5,"last update":"Fri Jan 29 13:57:31 2021","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":6.8769999999999998,"15min":3.1320000000000001},"min":{"1min":0.88500000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.3460000000000001,"5min":503.93000000000001,"15min":503.93000000000001},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.0269999999999999,"5min":6.7670000000000003,"15min":3.0150000000000001},"min":{"1min":0.83999999999999997,"5min":0.751,"15min":0.751},"max":{"1min":1.2569999999999999,"5min":503.74299999999999,"15min":503.74299999999999},"last":0.96199999999999997}]},{"osd":6,"last update":"Fri Jan 29 13:57:31 2021","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.4279999999999999,"15min":1.3440000000000001},"min":{"1min":1.0249999999999999,"5min":0.751,"15min":0.751},"max":{"1min":1.4710000000000001,"5min":4.7539999999999996,"15min":4.7539999999999996},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.141,"5min":10.393000000000001,"15min":34.238},"min":{"1min":0.876,"5min":0.876,"15min":0.876},"max":{"1min":1.3899999999999999,"5min":851.89099999999996,"15min":851.89099999999996},"last":1.1419999999999999}]},{"osd":7,"last update":"Fri Jan 29 13:57:31 2021","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.514,"15min":1.4910000000000001},"min":{"1min":1.0109999999999999,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.629,"5min":6.4240000000000004,"15min":6.4240000000000004},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.6830000000000001,"15min":1.917},"min":{"1min":1.038,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.6639999999999999,"5min":15.597,"15min":15.597},"last":1.246}]}]},{"osd":4,"up_from":42,"seq":180388626490,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4411548,"kb_used_data":4364052,"kb_used_omap":0,"kb_used_meta":47487,"kb_avail":89960292,"statfs":{"total":96636764160,"available":92119339008,"internally_reserved":0,"allocated":4468789248,"data_stored":5613909741,"data_compressed":118370768,"data_compressed_allocated":1155686400,"data_compressed_original":2311372800,"omap_allocated":972,"internal_metadata":48626740},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":1.095}]},{"osd":1,"last update":"Fri Jan 29 13:57:19 2021","interfaces":[{"interface":"back","average":{"1min":9.0939999999999994,"5min":31.199000000000002,"15min":58.671999999999997},"min":{"1min":0.747,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":177.846,"5min":924.63300000000004,"15min":924.63300000000004},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.151,"5min":15.670999999999999,"15min":5.9130000000000003},"min":{"1min":0.79300000000000004,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.637,"5min":898.86900000000003,"15min":898.86900000000003},"last":1.4630000000000001}]},{"osd":2,"last update":"Fri Jan 29 13:57:19 2021","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":11.949,"15min":4.7149999999999999},"min":{"1min":0.71299999999999997,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.6020000000000001,"5min":857.78899999999999,"15min":857.78899999999999},"last":1.395},{"interface":"front","average":{"1min":0.95099999999999996,"5min":9.8019999999999996,"15min":4.0369999999999999},"min":{"1min":0.76300000000000001,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.198,"5min":779.38400000000001,"15min":779.38400000000001},"last":0.86399999999999999}]},{"osd":3,"last update":"Fri Jan 29 13:57:19 2021","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":9.625,"15min":13.391},"min":{"1min":0.69799999999999995,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.423,"5min":311.68299999999999,"15min":311.68299999999999},"last":1.032},{"interface":"front","average":{"1min":35.929000000000002,"5min":8.1660000000000004,"15min":3.5739999999999998},"min":{"1min":0.83999999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":765.98299999999995,"5min":765.98299999999995,"15min":765.98299999999995},"last":1.4319999999999999}]},{"osd":5,"last update":"Fri Jan 29 13:57:19 2021","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":3.7069999999999999,"15min":1.956},"min":{"1min":0.61899999999999999,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.238,"5min":239.99199999999999,"15min":239.99199999999999},"last":1.012},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.1639999999999999,"15min":1.1579999999999999},"min":{"1min":0.84499999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.464,"5min":3.1749999999999998,"15min":3.1749999999999998},"last":1.141}]},{"osd":6,"last update":"Fri Jan 29 13:57:19 2021","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.2370000000000001,"15min":1.2050000000000001},"min":{"1min":0.73599999999999999,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.657,"5min":5.5890000000000004,"15min":5.5890000000000004},"last":1.492},{"interface":"front","average":{"1min":1.113,"5min":1.2110000000000001,"15min":1.282},"min":{"1min":0.86599999999999999,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.6719999999999999,"5min":3.6880000000000002,"15min":3.6880000000000002},"last":1.1220000000000001}]},{"osd":7,"last update":"Fri Jan 29 13:57:19 2021","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.2869999999999999,"15min":1.2809999999999999},"min":{"1min":0.98299999999999998,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":1.5620000000000001,"5min":3.0169999999999999,"15min":3.0169999999999999},"last":1.236},{"interface":"front","average":{"1min":33.942,"5min":7.8579999999999997,"15min":3.5139999999999998},"min":{"1min":1.0109999999999999,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":719.03700000000003,"5min":719.03700000000003,"15min":719.03700000000003},"last":1.5209999999999999}]}]},{"osd":5,"up_from":15,"seq":64424509507,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":765928,"kb_used_data":751904,"kb_used_omap":0,"kb_used_meta":14016,"kb_avail":93605912,"statfs":{"total":96636764160,"available":95852453888,"internally_reserved":0,"allocated":769949696,"data_stored":1055829374,"data_compressed":29380114,"data_compressed_allocated":286253056,"data_compressed_original":572506112,"omap_allocated":0,"internal_metadata":14352384},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:57:39 2021","interfaces":[{"interface":"back","average":{"1min":24.538,"5min":24.538,"15min":24.538},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":474.77499999999998,"5min":474.77499999999998,"15min":474.77499999999998},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.98799999999999999}]},{"osd":1,"last update":"Fri Jan 29 13:57:21 2021","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.96799999999999997,"15min":0.87},"min":{"1min":0.58299999999999996,"5min":0.498,"15min":0.498},"max":{"1min":1.2350000000000001,"5min":3.1819999999999999,"15min":3.1819999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.1499999999999999,"5min":11.096,"15min":4.2889999999999997},"min":{"1min":0.84999999999999998,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.518,"5min":732.80700000000002,"15min":732.80700000000002},"last":0.91200000000000003}]},{"osd":2,"last update":"Fri Jan 29 13:57:21 2021","interfaces":[{"interface":"back","average":{"1min":12.396000000000001,"5min":10.401999999999999,"15min":4.2919999999999998},"min":{"1min":0.81599999999999995,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":225.11799999999999,"5min":649.82899999999995,"15min":649.82899999999995},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.125,"15min":1.056},"min":{"1min":0.81799999999999995,"5min":0.623,"15min":0.623},"max":{"1min":1.256,"5min":4.1799999999999997,"15min":4.1799999999999997},"last":1.2010000000000001}]},{"osd":3,"last update":"Fri Jan 29 13:57:21 2021","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":1.1160000000000001,"15min":1.1259999999999999},"min":{"1min":0.67700000000000005,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.4730000000000001,"5min":4.0890000000000004,"15min":4.0890000000000004},"last":0.86099999999999999},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.27,"15min":1.228},"min":{"1min":0.92300000000000004,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.7350000000000001,"5min":4.827,"15min":4.827},"last":0.95499999999999996}]},{"osd":4,"last update":"Fri Jan 29 13:57:18 2021","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":13.199,"15min":16.175999999999998},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.6000000000000001,"5min":514.74199999999996,"15min":514.74199999999996},"last":1.228},{"interface":"front","average":{"1min":39.335999999999999,"5min":8.8729999999999993,"15min":3.7469999999999999},"min":{"1min":0.83599999999999997,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":762.10699999999997,"5min":762.10699999999997,"15min":762.10699999999997},"last":1.1499999999999999}]},{"osd":6,"last update":"Fri Jan 29 13:57:21 2021","interfaces":[{"interface":"back","average":{"1min":37.015999999999998,"5min":16.678000000000001,"15min":6.4059999999999997},"min":{"1min":1.0900000000000001,"5min":0.622,"15min":0.622},"max":{"1min":714.803,"5min":714.803,"15min":714.803},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.4630000000000001,"15min":1.27},"min":{"1min":1.119,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.8320000000000001,"5min":4.8700000000000001,"15min":4.8700000000000001},"last":1.2669999999999999}]},{"osd":7,"last update":"Fri Jan 29 13:57:21 2021","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.423,"15min":1.5049999999999999},"min":{"1min":1.0409999999999999,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.851,"5min":9.0109999999999992,"15min":9.0109999999999992},"last":1.282},{"interface":"front","average":{"1min":1.419,"5min":1.6479999999999999,"15min":1.758},"min":{"1min":1.1319999999999999,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":1.96,"5min":9.593,"15min":9.593},"last":1.3340000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509507,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4180812,"kb_used_data":4117700,"kb_used_omap":0,"kb_used_meta":63104,"kb_avail":90191028,"statfs":{"total":96636764160,"available":92355612672,"internally_reserved":0,"allocated":4216524800,"data_stored":4766177222,"data_compressed":58393750,"data_compressed_allocated":570441728,"data_compressed_original":1140883456,"omap_allocated":0,"internal_metadata":64618496},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:57:35 2021","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.5740000000000001},"last":1.1299999999999999},{"interface":"front","average":{"1min":31.253,"5min":31.253,"15min":31.253},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":729.27700000000004,"5min":729.27700000000004,"15min":729.27700000000004},"last":0.96599999999999997}]},{"osd":1,"last update":"Fri Jan 29 13:57:26 2021","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":8.0850000000000009,"15min":26.884},"min":{"1min":0.626,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.5029999999999999,"5min":684.42700000000002,"15min":684.42700000000002},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.26,"15min":1.1619999999999999},"min":{"1min":0.69699999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.319,"5min":6.1360000000000001,"15min":6.1360000000000001},"last":1.157}]},{"osd":2,"last update":"Fri Jan 29 13:57:26 2021","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.167,"15min":1.234},"min":{"1min":0.91800000000000004,"5min":0.498,"15min":0.498},"max":{"1min":1.383,"5min":7.2329999999999997,"15min":7.2329999999999997},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.0349999999999999,"5min":8.3000000000000007,"15min":3.6179999999999999},"min":{"1min":0.60499999999999998,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.542,"5min":607.71000000000004,"15min":607.71000000000004},"last":0.86699999999999999}]},{"osd":3,"last update":"Fri Jan 29 13:57:26 2021","interfaces":[{"interface":"back","average":{"1min":29.053999999999998,"5min":6.9160000000000004,"15min":3.1040000000000001},"min":{"1min":0.77700000000000002,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":501.613,"5min":501.613,"15min":501.613},"last":1.325},{"interface":"front","average":{"1min":1.1919999999999999,"5min":6.8719999999999999,"15min":3.0489999999999999},"min":{"1min":0.88500000000000001,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.4199999999999999,"5min":480.87299999999999,"15min":480.87299999999999},"last":1.1819999999999999}]},{"osd":4,"last update":"Fri Jan 29 13:57:13 2021","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":4.6059999999999999,"15min":2.4380000000000002},"min":{"1min":0.71799999999999997,"5min":0.62,"15min":0.62},"max":{"1min":1.6439999999999999,"5min":263.584,"15min":263.584},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.238,"15min":1.3939999999999999},"min":{"1min":0.82699999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.6200000000000001,"5min":7.3449999999999998,"15min":7.3449999999999998},"last":1.423}]},{"osd":5,"last update":"Fri Jan 29 13:57:26 2021","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.27,"15min":1.3400000000000001},"min":{"1min":1.0049999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.6020000000000001,"5min":4.0069999999999997,"15min":4.0069999999999997},"last":1.27},{"interface":"front","average":{"1min":1.25,"5min":1.569,"15min":1.492},"min":{"1min":1.012,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":1.5409999999999999,"5min":7.3140000000000001,"15min":7.3140000000000001},"last":1.5940000000000001}]},{"osd":7,"last update":"Fri Jan 29 13:57:26 2021","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.4159999999999999,"15min":1.3380000000000001},"min":{"1min":0.91100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.581,"5min":10.795,"15min":10.795},"last":1.655},{"interface":"front","average":{"1min":1.278,"5min":1.365,"15min":1.3839999999999999},"min":{"1min":0.99299999999999999,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.722,"5min":4.0579999999999998,"15min":4.0579999999999998},"last":1.383}]}]}]}} , 2021-01-29T13:57:47.178 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-29T13:57:47.465 INFO:teuthology.orchestra.run.smithi183.stdout: 2021-01-30T06:14:57.526 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T13:58:10.671 INFO:teuthology.orchestra.run.smithi183.stdout:} , 2021-01-29T13:58:10.682 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 , instructed osd(s) 2 to deep-scrub 2021-01-30T06:14:57.527 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 179, in _raise_for_status , raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 10000 --objects 6600 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 1200 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2021-01-30T06:14:57.527 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T13:59:00.217 DEBUG:teuthology.orchestra.run.smithi183:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- , 2021-01-29T14:01:20.328 INFO:tasks.ceph:Archiving logs... , 2021-01-29T14:01:20.329 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/log/ceph to /home/teuthworker/archive/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840036/remote/smithi183/log 2021-01-30T06:14:57.528 WARNING:scrape:Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 368, in archive , yield 2021-01-30T06:14:57.750 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-297-g27dc1eb3 (27dc1eb32728beb0c69aa65131bc609ef680c0aa) quincy (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f30135d315a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x55fd1cf11fe6] 2021-01-30T06:14:57.752 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T13:55:49.136 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument , 2021-01-29T13:55:49.136 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. , 2021-01-29T13:55:49.151 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2021-01-30T06:14:57.752 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T13:56:38.516 INFO:teuthology.orchestra.run.smithi183.stdout:oload 1000 , 2021-01-29T13:56:38.516 INFO:teuthology.orchestra.run.smithi183.stdout:max_change 3 , 2021-01-29T13:56:38.517 INFO:teuthology.orchestra.run.smithi183.stdout:max_change_osds 4 2021-01-30T06:14:57.755 WARNING:scrape:Ignoring malparsed backtrace: set osd.3 primary-affinity to 1 (8655362) , 2021-01-29T13:56:51.495 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 , 2021-01-29T13:56:52.480+0000 7f8f78ab0700 -1 osd.2 132 handle_osd_map injecting map crc failure 2021-01-30T06:14:57.761 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T13:57:08.429 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":278,"stamp":"2021-01-29T13:57:06.667864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20305121604,"num_objects":6822,"num_object_clones":0,"num_object_copies":19368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2076,"num_objects_unfound":0,"num_objects_dirty":4824,"num_whiteouts":0,"num_read":9651,"num_read_kb":4830,"num_write":25509,"num_write_kb":14727220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2477,"num_bytes_recovered":5602499186,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":21,"num_bytes_hit_set_archive":12196,"num_flush":2862,"num_flush_kb":8348668,"num_evict":884,"num_evict_kb":2571590,"num_promote":4835,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8249,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15129,"ondisk_log_size":15129,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":45,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":24946968,"kb_used_data":24625140,"kb_used_omap":1,"kb_used_meta":321598,"kb_avail":730027752,"statfs":{"total":773094113280,"available":747548418048,"internally_reserved":0,"allocated":25216143360,"data_stored":30051268153,"data_compressed":504955484,"data_compressed_allocated":4927905792,"data_compressed_original":9855811584,"omap_allocated":1944,"internal_metadata":329316456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-170,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":394035071,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8249,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.133172"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":"145","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.847683+0000","last_change":"2021-01-29T13:55:57.162929+0000","last_active":"2021-01-29T13:57:00.847683+0000","last_peered":"2021-01-29T13:57:00.847683+0000","last_clean":"2021-01-29T13:57:00.847683+0000","last_became_active":"2021-01-29T13:52:24.188931+0000","last_became_peered":"2021-01-29T13:52:24.188931+0000","last_unstale":"2021-01-29T13:57:00.847683+0000","last_undegraded":"2021-01-29T13:57:00.847683+0000","last_fullsized":"2021-01-29T13:57:00.847683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:55:57.162808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:55:57.162808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"120'6449","reported_seq":"8595","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.652824+0000","last_change":"2021-01-29T13:55:45.625490+0000","last_active":"2021-01-29T13:57:00.652824+0000","last_peered":"2021-01-29T13:57:00.652824+0000","last_clean":"2021-01-29T13:57:00.652824+0000","last_became_active":"2021-01-29T13:52:39.830011+0000","last_became_peered":"2021-01-29T13:52:39.830011+0000","last_unstale":"2021-01-29T13:57:00.652824+0000","last_undegraded":"2021-01-29T13:57:00.652824+0000","last_fullsized":"2021-01-29T13:57:00.652824+0000","mapping_epoch":26,"log_start":"72'3403","ondisk_log_start":"72'3403","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'6250","last_scrub_stamp":"2021-01-29T13:55:45.625379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:38.811171+0000","last_clean_scrub_stamp":"2021-01-29T13:55:45.625379+0000","log_size":3046,"ondisk_log_size":3046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2928382292,"num_objects":986,"num_object_clones":0,"num_object_copies":1972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":2501,"num_read_kb":1252,"num_write":5879,"num_write_kb":1639641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":3315,"num_flush":761,"num_flush_kb":2192275,"num_evict":269,"num_evict_kb":760560,"num_promote":1249,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"112","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.552366+0000","last_change":"2021-01-29T13:55:42.661441+0000","last_active":"2021-01-29T13:57:00.552366+0000","last_peered":"2021-01-29T13:57:00.552366+0000","last_clean":"2021-01-29T13:57:00.552366+0000","last_became_active":"2021-01-29T13:53:29.493255+0000","last_became_peered":"2021-01-29T13:53:29.493255+0000","last_unstale":"2021-01-29T13:57:00.552366+0000","last_undegraded":"2021-01-29T13:57:00.552366+0000","last_fullsized":"2021-01-29T13:57:00.552366+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:55:42.661371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:55:42.661371+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3","version":"120'671","reported_seq":"1016","reported_epoch":"142","state":"active+remapped+backfilling","last_fresh":"2021-01-29T13:57:05.597246+0000","last_change":"2021-01-29T13:56:47.680232+0000","last_active":"2021-01-29T13:57:05.597246+0000","last_peered":"2021-01-29T13:57:05.597246+0000","last_clean":"2021-01-29T13:56:36.386610+0000","last_became_active":"2021-01-29T13:56:47.674484+0000","last_became_peered":"2021-01-29T13:56:47.674484+0000","last_unstale":"2021-01-29T13:57:05.597246+0000","last_undegraded":"2021-01-29T13:57:05.597246+0000","last_fullsized":"2021-01-29T13:57:05.597246+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'570","last_scrub_stamp":"2021-01-29T13:55:44.911108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:35.792587+0000","last_clean_scrub_stamp":"2021-01-29T13:55:44.911108+0000","log_size":671,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2038945493,"num_objects":671,"num_object_clones":0,"num_object_copies":2684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":590,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":671,"num_write_kb":1991482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":601,"num_bytes_recovered":1847436236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4,7],"acting":[6,2,4,7],"avail_no_missing":["6(0)","2(1)","4(2)","7(3)"],"object_location_counts":[{"shards":"2(1),4(2),6(0),7(3)","objects":671}],"blocked_by":[],"up_primary":0,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"142","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:01.550240+0000","last_change":"2021-01-29T13:57:01.550191+0000","last_active":"2021-01-29T13:57:01.550240+0000","last_peered":"2021-01-29T13:57:01.550240+0000","last_clean":"2021-01-29T13:57:01.550240+0000","last_became_active":"2021-01-29T13:52:23.765595+0000","last_became_peered":"2021-01-29T13:52:23.765595+0000","last_unstale":"2021-01-29T13:57:01.550240+0000","last_undegraded":"2021-01-29T13:57:01.550240+0000","last_fullsized":"2021-01-29T13:57:01.550240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:57:01.550087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:57:01.550087+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"117'710","reported_seq":"880","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.847676+0000","last_change":"2021-01-29T13:56:09.328006+0000","last_active":"2021-01-29T13:57:00.847676+0000","last_peered":"2021-01-29T13:57:00.847676+0000","last_clean":"2021-01-29T13:57:00.847676+0000","last_became_active":"2021-01-29T13:53:09.969063+0000","last_became_peered":"2021-01-29T13:53:09.969063+0000","last_unstale":"2021-01-29T13:57:00.847676+0000","last_undegraded":"2021-01-29T13:57:00.847676+0000","last_fullsized":"2021-01-29T13:57:00.847676+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'710","last_scrub_stamp":"2021-01-29T13:56:09.327903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:35.792587+0000","last_clean_scrub_stamp":"2021-01-29T13:56:09.327903+0000","log_size":710,"ondisk_log_size":710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2107977089,"num_objects":710,"num_object_clones":0,"num_object_copies":2840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":710,"num_write_kb":2058922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2,4],"acting":[5,7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"155","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.849258+0000","last_change":"2021-01-29T13:56:59.467725+0000","last_active":"2021-01-29T13:57:00.849258+0000","last_peered":"2021-01-29T13:57:00.849258+0000","last_clean":"2021-01-29T13:57:00.849258+0000","last_became_active":"2021-01-29T13:52:24.188578+0000","last_became_peered":"2021-01-29T13:52:24.188578+0000","last_unstale":"2021-01-29T13:57:00.849258+0000","last_undegraded":"2021-01-29T13:57:00.849258+0000","last_fullsized":"2021-01-29T13:57:00.849258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:56:59.467654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:56:59.467654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"142'4","reported_seq":"152","reported_epoch":"142","state":"active+clean","last_fresh":"2021-01-29T13:57:01.982145+0000","last_change":"2021-01-29T13:57:01.982145+0000","last_active":"2021-01-29T13:57:01.982145+0000","last_peered":"2021-01-29T13:57:01.982145+0000","last_clean":"2021-01-29T13:57:01.982145+0000","last_became_active":"2021-01-29T13:52:20.655015+0000","last_became_peered":"2021-01-29T13:52:20.655015+0000","last_unstale":"2021-01-29T13:57:01.982145+0000","last_undegraded":"2021-01-29T13:57:01.982145+0000","last_fullsized":"2021-01-29T13:57:01.982145+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'4","last_scrub_stamp":"2021-01-29T13:57:01.982073+0000","last_deep_scrub":"142'4","last_deep_scrub_stamp":"2021-01-29T13:57:01.982073+0000","last_clean_scrub_stamp":"2021-01-29T13:57:01.982073+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8249,"num_omap_keys":2,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"120'761","reported_seq":"2454","reported_epoch":"142","state":"active+remapped+backfilling","last_fresh":"2021-01-29T13:57:03.004043+0000","last_change":"2021-01-29T13:56:48.363777+0000","last_active":"2021-01-29T13:57:03.004043+0000","last_peered":"2021-01-29T13:57:03.004043+0000","last_clean":"2021-01-29T13:56:36.386422+0000","last_became_active":"2021-01-29T13:56:48.362095+0000","last_became_peered":"2021-01-29T13:56:48.362095+0000","last_unstale":"2021-01-29T13:57:03.004043+0000","last_undegraded":"2021-01-29T13:57:03.004043+0000","last_fullsized":"2021-01-29T13:57:03.004043+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'761","last_scrub_stamp":"2021-01-29T13:55:53.981049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:35.792587+0000","last_clean_scrub_stamp":"2021-01-29T13:55:53.981049+0000","log_size":761,"ondisk_log_size":761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2244508217,"num_objects":761,"num_object_clones":0,"num_object_copies":3044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":637,"num_objects_unfound":0,"num_objects_dirty":761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":761,"num_write_kb":2192275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":747,"num_bytes_recovered":2213000897,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7,0],"acting":[4,2,7,1],"avail_no_missing":["4(0)","1(3)","2(1)","7(2)"],"object_location_counts":[{"shards":"1(3),2(1),4(0),7(2)","objects":761}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"144","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.849237+0000","last_change":"2021-01-29T13:56:41.371116+0000","last_active":"2021-01-29T13:57:00.849237+0000","last_peered":"2021-01-29T13:57:00.849237+0000","last_clean":"2021-01-29T13:57:00.849237+0000","last_became_active":"2021-01-29T13:52:23.765671+0000","last_became_peered":"2021-01-29T13:52:23.765671+0000","last_unstale":"2021-01-29T13:57:00.849237+0000","last_undegraded":"2021-01-29T13:57:00.849237+0000","last_fullsized":"2021-01-29T13:57:00.849237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:56:41.371025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:56:41.371025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"118'720","reported_seq":"886","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.652827+0000","last_change":"2021-01-29T13:55:59.202455+0000","last_active":"2021-01-29T13:57:00.652827+0000","last_peered":"2021-01-29T13:57:00.652827+0000","last_clean":"2021-01-29T13:57:00.652827+0000","last_became_active":"2021-01-29T13:53:09.587252+0000","last_became_peered":"2021-01-29T13:53:09.587252+0000","last_unstale":"2021-01-29T13:57:00.652827+0000","last_undegraded":"2021-01-29T13:57:00.652827+0000","last_fullsized":"2021-01-29T13:57:00.652827+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'720","last_scrub_stamp":"2021-01-29T13:55:59.202349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:35.792587+0000","last_clean_scrub_stamp":"2021-01-29T13:55:59.202349+0000","log_size":720,"ondisk_log_size":720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2156154079,"num_objects":720,"num_object_clones":0,"num_object_copies":2880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":720,"num_write_kb":2105989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4,3],"acting":[6,2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"120'5986","reported_seq":"10252","reported_epoch":"142","state":"active+remapped+backfilling","last_fresh":"2021-01-29T13:57:05.668638+0000","last_change":"2021-01-29T13:56:47.680268+0000","last_active":"2021-01-29T13:57:05.668638+0000","last_peered":"2021-01-29T13:57:05.668638+0000","last_clean":"2021-01-29T13:56:36.386614+0000","last_became_active":"2021-01-29T13:56:47.673885+0000","last_became_peered":"2021-01-29T13:56:47.673885+0000","last_unstale":"2021-01-29T13:57:05.668638+0000","last_undegraded":"2021-01-29T13:57:05.668638+0000","last_fullsized":"2021-01-29T13:57:05.668638+0000","mapping_epoch":127,"log_start":"71'2904","ondisk_log_start":"71'2904","created":26,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'5955","last_scrub_stamp":"2021-01-29T13:55:51.275882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:38.811171+0000","last_clean_scrub_stamp":"2021-01-29T13:55:51.275882+0000","log_size":3082,"ondisk_log_size":3082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2947693900,"num_objects":985,"num_object_clones":0,"num_object_copies":1970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":849,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":2316,"num_read_kb":1158,"num_write":5435,"num_write_kb":1544566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1129,"num_bytes_recovered":1542062053,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":2771,"num_flush":671,"num_flush_kb":1991482,"num_evict":178,"num_evict_kb":526959,"num_promote":1172,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":985}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"157","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:01.451085+0000","last_change":"2021-01-29T13:57:01.451085+0000","last_active":"2021-01-29T13:57:01.451085+0000","last_peered":"2021-01-29T13:57:01.451085+0000","last_clean":"2021-01-29T13:57:01.451085+0000","last_became_active":"2021-01-29T13:53:09.377717+0000","last_became_peered":"2021-01-29T13:53:09.377717+0000","last_unstale":"2021-01-29T13:57:01.451085+0000","last_undegraded":"2021-01-29T13:57:01.451085+0000","last_fullsized":"2021-01-29T13:57:01.451085+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:55:33.131622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:55:33.131622+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"140","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.847720+0000","last_change":"2021-01-29T13:56:53.807344+0000","last_active":"2021-01-29T13:57:00.847720+0000","last_peered":"2021-01-29T13:57:00.847720+0000","last_clean":"2021-01-29T13:57:00.847720+0000","last_became_active":"2021-01-29T13:52:40.828334+0000","last_became_peered":"2021-01-29T13:52:40.828334+0000","last_unstale":"2021-01-29T13:57:00.847720+0000","last_undegraded":"2021-01-29T13:57:00.847720+0000","last_fullsized":"2021-01-29T13:57:00.847720+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:56:53.807251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:56:53.807251+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"117'6273","reported_seq":"8319","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.552349+0000","last_change":"2021-01-29T13:55:55.445357+0000","last_active":"2021-01-29T13:57:00.552349+0000","last_peered":"2021-01-29T13:57:00.552349+0000","last_clean":"2021-01-29T13:57:00.552349+0000","last_became_active":"2021-01-29T13:53:09.369472+0000","last_became_peered":"2021-01-29T13:53:09.369472+0000","last_unstale":"2021-01-29T13:57:00.552349+0000","last_undegraded":"2021-01-29T13:57:00.552349+0000","last_fullsized":"2021-01-29T13:57:00.552349+0000","mapping_epoch":42,"log_start":"71'3203","ondisk_log_start":"71'3203","created":26,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'6273","last_scrub_stamp":"2021-01-29T13:55:55.445229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:38.811171+0000","last_clean_scrub_stamp":"2021-01-29T13:55:55.445229+0000","log_size":3070,"ondisk_log_size":3070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2935339657,"num_objects":992,"num_object_clones":0,"num_object_copies":1984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":496,"num_whiteouts":0,"num_read":2414,"num_read_kb":1208,"num_write":5647,"num_write_kb":1593121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":3044,"num_flush":710,"num_flush_kb":2058922,"num_evict":219,"num_evict_kb":640347,"num_promote":1206,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"24","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.611252+0000","last_change":"2021-01-29T13:56:46.442031+0000","last_active":"2021-01-29T13:57:00.611252+0000","last_peered":"2021-01-29T13:57:00.611252+0000","last_clean":"2021-01-29T13:57:00.611252+0000","last_became_active":"2021-01-29T13:56:46.441589+0000","last_became_peered":"2021-01-29T13:56:46.441589+0000","last_unstale":"2021-01-29T13:57:00.611252+0000","last_undegraded":"2021-01-29T13:57:00.611252+0000","last_fullsized":"2021-01-29T13:57:00.611252+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:55:55.134762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:55:55.134762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"118'6270","reported_seq":"8351","reported_epoch":"142","state":"active+clean","last_fresh":"2021-01-29T13:57:04.235378+0000","last_change":"2021-01-29T13:57:04.235378+0000","last_active":"2021-01-29T13:57:04.235378+0000","last_peered":"2021-01-29T13:57:04.235378+0000","last_clean":"2021-01-29T13:57:04.235378+0000","last_became_active":"2021-01-29T13:52:39.823642+0000","last_became_peered":"2021-01-29T13:52:39.823642+0000","last_unstale":"2021-01-29T13:57:04.235378+0000","last_undegraded":"2021-01-29T13:57:04.235378+0000","last_fullsized":"2021-01-29T13:57:04.235378+0000","mapping_epoch":26,"log_start":"71'3205","ondisk_log_start":"71'3205","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6270","last_scrub_stamp":"2021-01-29T13:57:04.235268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:38.811171+0000","last_clean_scrub_stamp":"2021-01-29T13:57:04.235268+0000","log_size":3065,"ondisk_log_size":3065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2946120877,"num_objects":995,"num_object_clones":0,"num_object_copies":1990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":2420,"num_read_kb":1212,"num_write":5684,"num_write_kb":1601214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":3066,"num_flush":720,"num_flush_kb":2105989,"num_evict":218,"num_evict_kb":643724,"num_promote":1208,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":4,"stat_sum":{"num_bytes":11757536726,"num_objects":3958,"num_object_clones":0,"num_object_copies":7916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":849,"num_objects_unfound":0,"num_objects_dirty":1960,"num_whiteouts":0,"num_read":9651,"num_read_kb":4830,"num_write":22645,"num_write_kb":6378542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1129,"num_bytes_recovered":1542062053,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":21,"num_bytes_hit_set_archive":12196,"num_flush":2862,"num_flush_kb":8348668,"num_evict":884,"num_evict_kb":2571590,"num_promote":4835,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11918577664,"data_stored":11828415316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12263,"ondisk_log_size":12263,"up":8,"acting":8,"num_store_stats":6},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":8547584878,"num_objects":2862,"num_object_clones":0,"num_object_copies":11448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1227,"num_objects_unfound":0,"num_objects_dirty":2862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2862,"num_write_kb":8348668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1348,"num_bytes_recovered":4060437133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13299490816,"data_stored":18228555776,"data_compressed":505072361,"data_compressed_allocated":4929064960,"data_compressed_original":9858129920,"omap_allocated":0,"internal_metadata":0},"log_size":2862,"ondisk_log_size":2862,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8249,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":0,"up_from":124,"seq":532575944712,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1942776,"kb_used_data":1921328,"kb_used_omap":0,"kb_used_meta":21439,"kb_avail":92429064,"statfs":{"total":96636764160,"available":94647361536,"internally_reserved":0,"allocated":1967439872,"data_stored":2257930354,"data_compressed":30754208,"data_compressed_allocated":299577344,"data_compressed_original":599154688,"omap_allocated":972,"internal_metadata":21953588},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.23}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98799999999999999}]}]},{"osd":7,"up_from":15,"seq":64424509501,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2311348,"kb_used_data":2281004,"kb_used_omap":0,"kb_used_meta":30336,"kb_avail":92060492,"statfs":{"total":96636764160,"available":94269943808,"internally_reserved":0,"allocated":2335748096,"data_stored":3200465622,"data_compressed":88587048,"data_compressed_allocated":865042432,"data_compressed_original":1730084864,"omap_allocated":0,"internal_metadata":31064064},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:54:26 2021","interfaces":[{"interface":"back","average":{"1min":51.597999999999999,"5min":29.779,"15min":26.143000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":906.00800000000004,"5min":906.00800000000004,"15min":906.00800000000004},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.1040000000000001,"5min":0.95499999999999996,"15min":0.93000000000000005},"min":{"1min":0.61499999999999999,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":4.0259999999999998,"5min":4.0259999999999998,"15min":4.0259999999999998},"last":1.3089999999999999}]},{"osd":1,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.3340000000000001,"15min":1.49},"min":{"1min":0.65800000000000003,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.4410000000000001,"5min":11.195,"15min":11.195},"last":1.272},{"interface":"front","average":{"1min":1.006,"5min":23.977,"15min":32.523000000000003},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.3759999999999999,"5min":777.02499999999998,"15min":777.02499999999998},"last":1.581}]},{"osd":2,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":46.633000000000003,"5min":10.444000000000001,"15min":4.5460000000000003},"min":{"1min":0.62,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":512.423,"5min":512.423,"15min":512.423},"last":0.999},{"interface":"front","average":{"1min":1.1559999999999999,"5min":2.1200000000000001,"15min":1.774},"min":{"1min":0.79300000000000004,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.472,"5min":60.529000000000003,"15min":60.529000000000003},"last":1.3759999999999999}]},{"osd":3,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":10.481,"5min":3.0230000000000001,"15min":1.746},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":211.47900000000001,"5min":211.47900000000001,"15min":211.47900000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.46,"15min":1.607},"min":{"1min":0.92300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.4930000000000001,"5min":11.146000000000001,"15min":11.146000000000001},"last":1.4079999999999999}]},{"osd":4,"last update":"Fri Jan 29 13:56:16 2021","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.264,"15min":1.3380000000000001},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.4299999999999999,"5min":5.8449999999999998,"15min":5.8449999999999998},"last":1.343},{"interface":"front","average":{"1min":1.0229999999999999,"5min":14.401999999999999,"15min":20.280000000000001},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.4399999999999999,"5min":391.66699999999997,"15min":391.66699999999997},"last":1.54}]},{"osd":5,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":3.4470000000000001,"15min":2.2829999999999999},"min":{"1min":0.74199999999999999,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.601,"5min":175.83199999999999,"15min":175.83199999999999},"last":1.625},{"interface":"front","average":{"1min":1.105,"5min":1.2549999999999999,"15min":1.2},"min":{"1min":0.77600000000000002,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.526,"5min":4.4020000000000001,"15min":4.4020000000000001},"last":1.6020000000000001}]},{"osd":6,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.5649999999999999,"15min":1.536},"min":{"1min":0.84199999999999997,"5min":0.754,"15min":0.754},"max":{"1min":1.5269999999999999,"5min":10.063000000000001,"15min":10.063000000000001},"last":1.647},{"interface":"front","average":{"1min":1.123,"5min":14.808,"15min":27.946000000000002},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.4690000000000001,"5min":623.01999999999998,"15min":623.01999999999998},"last":1.754}]}]},{"osd":2,"up_from":15,"seq":64424509500,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5736668,"kb_used_data":5682580,"kb_used_omap":0,"kb_used_meta":54080,"kb_avail":88635172,"statfs":{"total":96636764160,"available":90762416128,"internally_reserved":0,"allocated":5818961920,"data_stored":6956451332,"data_compressed":118712623,"data_compressed_allocated":1158184960,"data_compressed_original":2316369920,"omap_allocated":0,"internal_metadata":55377920},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:54:22 2021","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":29.140000000000001,"15min":33.799999999999997},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":2.012,"5min":565.52099999999996,"15min":565.52099999999996},"last":0.997},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.0209999999999999,"15min":0.94299999999999995},"min":{"1min":0.72499999999999998,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":3.6030000000000002,"5min":3.6030000000000002,"15min":3.6030000000000002},"last":0.77000000000000002}]},{"osd":1,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":40.582999999999998,"5min":17.475000000000001,"15min":6.4610000000000003},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":676.39099999999996,"5min":721.11000000000001,"15min":721.11000000000001},"last":1.0489999999999999},{"interface":"front","average":{"1min":0.89700000000000002,"5min":21.452999999999999,"15min":7.8179999999999996},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.1479999999999999,"5min":1000.376,"15min":1000.376},"last":0.81299999999999994}]},{"osd":3,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":51.539000000000001,"5min":11.381,"15min":4.3440000000000003},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":501.61200000000002,"5min":501.61200000000002,"15min":501.61200000000002},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.85399999999999998,"5min":1.1890000000000001,"15min":1.0640000000000001},"min":{"1min":0.57399999999999995,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.6839999999999999,"5min":6.1870000000000003,"15min":6.1870000000000003},"last":0.78900000000000003}]},{"osd":4,"last update":"Fri Jan 29 13:56:15 2021","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.478,"15min":1.53},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.6719999999999999,"5min":3.694,"15min":3.694},"last":1.6160000000000001},{"interface":"front","average":{"1min":1.155,"5min":1.407,"15min":1.3859999999999999},"min":{"1min":0.871,"5min":0.871,"15min":0.871},"max":{"1min":1.7509999999999999,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":1.6479999999999999}]},{"osd":5,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":21.327999999999999,"5min":5.8109999999999999,"15min":2.7749999999999999},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":183.761,"5min":183.761,"15min":183.761},"last":0.86199999999999999},{"interface":"front","average":{"1min":41.545999999999999,"5min":9.516,"15min":4.0499999999999998},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":521.02499999999998,"5min":521.02499999999998,"15min":521.02499999999998},"last":1.0289999999999999}]},{"osd":6,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.3720000000000001,"15min":1.3009999999999999},"min":{"1min":0.89900000000000002,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.474,"5min":3.46,"15min":3.46},"last":1.544},{"interface":"front","average":{"1min":1.111,"5min":1.4450000000000001,"15min":1.2809999999999999},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.4570000000000001,"5min":7.0209999999999999,"15min":7.0209999999999999},"last":0.88200000000000001}]},{"osd":7,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":8.5039999999999996,"15min":3.754},"min":{"1min":0.99099999999999999,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":1.7050000000000001,"5min":591.18899999999996,"15min":591.18899999999996},"last":1.5820000000000001},{"interface":"front","average":{"1min":52.493000000000002,"5min":11.779999999999999,"15min":4.8220000000000001},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":923.40099999999995,"5min":923.40099999999995,"15min":923.40099999999995},"last":0.95499999999999996}]}]},{"osd":1,"up_from":15,"seq":64424509501,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2154844,"kb_used_data":2115092,"kb_used_omap":0,"kb_used_meta":39744,"kb_avail":92216996,"statfs":{"total":96636764160,"available":94430203904,"internally_reserved":0,"allocated":2165854208,"data_stored":2457969146,"data_compressed":31021897,"data_compressed_allocated":302632960,"data_compressed_original":605265920,"omap_allocated":0,"internal_metadata":40697856},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:54:25 2021","interfaces":[{"interface":"back","average":{"1min":1.7629999999999999,"5min":1.0349999999999999,"15min":0.91300000000000003},"min":{"1min":0.63300000000000001,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":5.0629999999999997,"5min":5.0629999999999997,"15min":5.0629999999999997},"last":0.92400000000000004},{"interface":"front","average":{"1min":39.090000000000003,"5min":8.5060000000000002,"15min":3.4079999999999999},"min":{"1min":0.58699999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":752.73400000000004,"5min":752.73400000000004,"15min":752.73400000000004},"last":0.54300000000000004}]},{"osd":2,"last update":"Fri Jan 29 13:56:29 2021","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":22.648,"15min":29.776},"min":{"1min":0.55100000000000005,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":4.2629999999999999,"5min":832.21600000000001,"15min":832.21600000000001},"last":0.83499999999999996},{"interface":"front","average":{"1min":5.2699999999999996,"5min":11.266,"15min":4.4199999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":84.277000000000001,"5min":899.43399999999997,"15min":899.43399999999997},"last":0.86599999999999999}]},{"osd":3,"last update":"Fri Jan 29 13:56:29 2021","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":1.306,"15min":1.099},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":2.1499999999999999,"5min":6.0579999999999998,"15min":6.0579999999999998},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":5.548,"15min":9.8919999999999995},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.5489999999999999,"5min":244.60900000000001,"15min":244.60900000000001},"last":0.68899999999999995}]},{"osd":4,"last update":"Fri Jan 29 13:56:15 2021","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.4610000000000001,"15min":1.633},"min":{"1min":0.63800000000000001,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.5960000000000001,"5min":5.242,"15min":5.242},"last":0.71399999999999997},{"interface":"front","average":{"1min":1.202,"5min":1.9259999999999999,"15min":2.1680000000000001},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.5760000000000001,"5min":6.2590000000000003,"15min":6.2590000000000003},"last":1.024}]},{"osd":5,"last update":"Fri Jan 29 13:56:29 2021","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":14.667999999999999,"15min":5.694},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":2.0350000000000001,"5min":701.73400000000004,"15min":701.73400000000004},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.133,"5min":1.3400000000000001,"15min":1.26},"min":{"1min":0.75700000000000001,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.917,"5min":5.758,"15min":5.758},"last":0.76600000000000001}]},{"osd":6,"last update":"Fri Jan 29 13:56:29 2021","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.4419999999999999,"15min":1.3440000000000001},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.9139999999999999,"5min":5.7199999999999998,"15min":5.7199999999999998},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.258,"5min":4.0110000000000001,"15min":6.2149999999999999},"min":{"1min":0.72299999999999998,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.6230000000000002,"5min":132.35300000000001,"15min":132.35300000000001},"last":1.0589999999999999}]},{"osd":7,"last update":"Fri Jan 29 13:56:29 2021","interfaces":[{"interface":"back","average":{"1min":37.829999999999998,"5min":8.8330000000000002,"15min":3.823},"min":{"1min":0.81299999999999994,"5min":0.752,"15min":0.752},"max":{"1min":696.96400000000006,"5min":696.96400000000006,"15min":696.96400000000006},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.4990000000000001,"15min":1.413},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.0369999999999999,"5min":5.9450000000000003,"15min":5.9450000000000003},"last":0.85199999999999998}]}]},{"osd":3,"up_from":15,"seq":64424509501,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3443244,"kb_used_data":3391652,"kb_used_omap":0,"kb_used_meta":51584,"kb_avail":90928596,"statfs":{"total":96636764160,"available":93110882304,"internally_reserved":0,"allocated":3473051648,"data_stored":3742620723,"data_compressed":29735076,"data_compressed_allocated":290086912,"data_compressed_original":580173824,"omap_allocated":0,"internal_metadata":52822016},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:54:26 2021","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":0.93999999999999995,"15min":0.90400000000000003},"min":{"1min":0.67500000000000004,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.617,"5min":2.617,"15min":2.617},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.135,"5min":3.3719999999999999,"15min":3.7450000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":3.9620000000000002,"5min":60.329000000000001,"15min":60.329000000000001},"last":1.0529999999999999}]},{"osd":1,"last update":"Fri Jan 29 13:56:30 2021","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.97599999999999998,"15min":0.91300000000000003},"min":{"1min":0.497,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0109999999999999,"5min":2.5819999999999999,"15min":2.5819999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.82699999999999996,"5min":10.324,"15min":4.0970000000000004},"min":{"1min":0.63800000000000001,"5min":0.496,"15min":0.496},"max":{"1min":1.0629999999999999,"5min":977.78200000000004,"15min":977.78200000000004},"last":0.66900000000000004}]},{"osd":2,"last update":"Fri Jan 29 13:56:30 2021","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":1.208,"15min":1.0880000000000001},"min":{"1min":0.61099999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.22,"5min":5.7539999999999996,"15min":5.7539999999999996},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.018,"5min":1.383,"15min":1.4339999999999999},"min":{"1min":0.73199999999999998,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.4870000000000001,"5min":6.4509999999999996,"15min":6.4509999999999996},"last":0.96899999999999997}]},{"osd":4,"last update":"Fri Jan 29 13:56:11 2021","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":11.106,"15min":4.5620000000000003},"min":{"1min":0.70799999999999996,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.3979999999999999,"5min":916.09500000000003,"15min":916.09500000000003},"last":0.71099999999999997},{"interface":"front","average":{"1min":1.1479999999999999,"5min":11.696999999999999,"15min":4.8819999999999997},"min":{"1min":0.82099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.593,"5min":977.91499999999996,"15min":977.91499999999996},"last":0.92900000000000005}]},{"osd":5,"last update":"Fri Jan 29 13:56:30 2021","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":6.9089999999999998,"15min":3.1429999999999998},"min":{"1min":0.72799999999999998,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.8160000000000001,"5min":503.93000000000001,"15min":503.93000000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.077,"5min":6.79,"15min":3.0230000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.3799999999999999,"5min":503.74299999999999,"15min":503.74299999999999},"last":1.141}]},{"osd":6,"last update":"Fri Jan 29 13:56:30 2021","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.4490000000000001,"15min":1.351},"min":{"1min":0.874,"5min":0.751,"15min":0.751},"max":{"1min":3.2930000000000001,"5min":4.7539999999999996,"15min":4.7539999999999996},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.339,"5min":19.396999999999998,"15min":37.238999999999997},"min":{"1min":0.876,"5min":0.876,"15min":0.876},"max":{"1min":3.3690000000000002,"5min":851.89099999999996,"15min":851.89099999999996},"last":1.173}]},{"osd":7,"last update":"Fri Jan 29 13:56:30 2021","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.5609999999999999,"15min":1.5069999999999999},"min":{"1min":0.90800000000000003,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.613,"5min":6.4240000000000004,"15min":6.4240000000000004},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.8300000000000001,"15min":1.966},"min":{"1min":1.0629999999999999,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.635,"5min":15.597,"15min":15.597},"last":1.038}]}]},{"osd":4,"up_from":42,"seq":180388626482,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4411588,"kb_used_data":4363992,"kb_used_omap":0,"kb_used_meta":47423,"kb_avail":89960252,"statfs":{"total":96636764160,"available":92119298048,"internally_reserved":0,"allocated":4468727808,"data_stored":5613881037,"data_compressed":118370768,"data_compressed_allocated":1155686400,"data_compressed_original":2311372800,"omap_allocated":972,"internal_metadata":48561204},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:54:12 2021","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":4.2450000000000001,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.746},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":4.1589999999999998,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":1.141}]},{"osd":1,"last update":"Fri Jan 29 13:56:18 2021","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":43.862000000000002,"15min":62.893000000000001},"min":{"1min":0.76800000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":2.6850000000000001,"5min":924.63300000000004,"15min":924.63300000000004},"last":1.008},{"interface":"front","average":{"1min":74.046000000000006,"5min":15.648,"15min":5.9059999999999997},"min":{"1min":0.84299999999999997,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":898.86900000000003,"5min":898.86900000000003,"15min":898.86900000000003},"last":1.2090000000000001}]},{"osd":2,"last update":"Fri Jan 29 13:56:18 2021","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":11.952,"15min":4.7160000000000002},"min":{"1min":0.745,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.3149999999999999,"5min":857.78899999999999,"15min":857.78899999999999},"last":1.1679999999999999},{"interface":"front","average":{"1min":44.206000000000003,"5min":9.843,"15min":4.0510000000000002},"min":{"1min":0.68600000000000005,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":779.38400000000001,"5min":779.38400000000001,"15min":779.38400000000001},"last":0.78100000000000003}]},{"osd":3,"last update":"Fri Jan 29 13:56:18 2021","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":12.48,"15min":14.343},"min":{"1min":0.754,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":2.738,"5min":311.68299999999999,"15min":311.68299999999999},"last":1.079},{"interface":"front","average":{"1min":1.028,"5min":1.236,"15min":1.264},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.4359999999999999,"5min":3.6539999999999999,"15min":3.6539999999999999},"last":1.2509999999999999}]},{"osd":5,"last update":"Fri Jan 29 13:56:18 2021","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":3.7309999999999999,"15min":1.964},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.5109999999999999,"5min":239.99199999999999,"15min":239.99199999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.171,"15min":1.1599999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":3.1749999999999998,"5min":3.1749999999999998,"15min":3.1749999999999998},"last":1.03}]},{"osd":6,"last update":"Fri Jan 29 13:56:18 2021","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.238,"15min":1.206},"min":{"1min":0.72499999999999998,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.7789999999999999,"5min":5.5890000000000004,"15min":5.5890000000000004},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.252,"15min":1.296},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.657,"5min":3.6880000000000002,"15min":3.6880000000000002},"last":0.91600000000000004}]},{"osd":7,"last update":"Fri Jan 29 13:56:18 2021","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.3,"15min":1.286},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":2.988,"5min":3.0169999999999999,"15min":3.0169999999999999},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.3380000000000001,"15min":1.3400000000000001},"min":{"1min":0.96099999999999997,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":1.635,"5min":3.6699999999999999,"15min":3.6699999999999999},"last":1.357}]}]},{"osd":5,"up_from":15,"seq":64424509500,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":765812,"kb_used_data":751852,"kb_used_omap":0,"kb_used_meta":13952,"kb_avail":93606028,"statfs":{"total":96636764160,"available":95852572672,"internally_reserved":0,"allocated":769896448,"data_stored":1055801421,"data_compressed":29380114,"data_compressed_allocated":286253056,"data_compressed_original":572506112,"omap_allocated":0,"internal_metadata":14286848},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:54:20 2021","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.83999999999999997,"15min":0.81299999999999994},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":2.3250000000000002,"5min":2.3250000000000002,"15min":2.3250000000000002},"last":1.079},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.101,"15min":1.044},"min":{"1min":0.65600000000000003,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":4.8410000000000002,"5min":4.8410000000000002,"15min":4.8410000000000002},"last":1.306}]},{"osd":1,"last update":"Fri Jan 29 13:56:20 2021","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":0.93600000000000005,"15min":0.85999999999999999},"min":{"1min":0.68500000000000005,"5min":0.498,"15min":0.498},"max":{"1min":2.3690000000000002,"5min":3.1819999999999999,"15min":3.1819999999999999},"last":1.1140000000000001},{"interface":"front","average":{"1min":51.244999999999997,"5min":11.042999999999999,"15min":4.2709999999999999},"min":{"1min":0.82999999999999996,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":732.80700000000002,"5min":732.80700000000002,"15min":732.80700000000002},"last":1.365}]},{"osd":2,"last update":"Fri Jan 29 13:56:20 2021","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":8.1699999999999999,"15min":3.548},"min":{"1min":0.73599999999999999,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.161,"5min":649.82899999999995,"15min":649.82899999999995},"last":1.6950000000000001},{"interface":"front","average":{"1min":1.095,"5min":1.1180000000000001,"15min":1.054},"min":{"1min":0.70699999999999996,"5min":0.623,"15min":0.623},"max":{"1min":2.4169999999999998,"5min":4.1799999999999997,"15min":4.1799999999999997},"last":1.145}]},{"osd":3,"last update":"Fri Jan 29 13:56:20 2021","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":1.137,"15min":1.133},"min":{"1min":0.626,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.2829999999999999,"5min":4.0890000000000004,"15min":4.0890000000000004},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.274,"15min":1.23},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":2.1200000000000001,"5min":4.827,"15min":4.827},"last":1.7350000000000001}]},{"osd":4,"last update":"Fri Jan 29 13:56:17 2021","interfaces":[{"interface":"back","average":{"1min":28.036000000000001,"5min":16.495999999999999,"15min":17.274999999999999},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":514.74199999999996,"5min":514.74199999999996,"15min":514.74199999999996},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.2430000000000001,"15min":1.204},"min":{"1min":0.86699999999999999,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":2.266,"5min":4.4359999999999999,"15min":4.4359999999999999},"last":1.7689999999999999}]},{"osd":6,"last update":"Fri Jan 29 13:56:20 2021","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":9.5289999999999999,"15min":4.0229999999999997},"min":{"1min":0.81499999999999995,"5min":0.622,"15min":0.622},"max":{"1min":2.3039999999999998,"5min":520.71799999999996,"15min":520.71799999999996},"last":1.804},{"interface":"front","average":{"1min":1.345,"5min":1.421,"15min":1.256},"min":{"1min":0.84399999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.3929999999999998,"5min":4.8700000000000001,"15min":4.8700000000000001},"last":1.8320000000000001}]},{"osd":7,"last update":"Fri Jan 29 13:56:20 2021","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.4710000000000001,"15min":1.5209999999999999},"min":{"1min":0.92700000000000005,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.3420000000000001,"5min":9.0109999999999992,"15min":9.0109999999999992},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.383,"5min":1.7270000000000001,"15min":1.7849999999999999},"min":{"1min":1.0049999999999999,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":2.3319999999999999,"5min":9.593,"15min":9.593},"last":1.96}]}]},{"osd":6,"up_from":15,"seq":64424509500,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4180688,"kb_used_data":4117640,"kb_used_omap":0,"kb_used_meta":63040,"kb_avail":90191152,"statfs":{"total":96636764160,"available":92355739648,"internally_reserved":0,"allocated":4216463360,"data_stored":4766148518,"data_compressed":58393750,"data_compressed_allocated":570441728,"data_compressed_original":1140883456,"omap_allocated":0,"internal_metadata":64552960},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:54:20 2021","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.379,"15min":1.381},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":3.0899999999999999,"5min":5.8380000000000001,"15min":5.8380000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":1.286,"5min":1.012,"15min":0.96599999999999997},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":7.3810000000000002,"5min":7.3810000000000002,"15min":7.3810000000000002},"last":0.50800000000000001}]},{"osd":1,"last update":"Fri Jan 29 13:56:22 2021","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":15.146000000000001,"15min":29.238},"min":{"1min":0.65400000000000003,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.353,"5min":684.42700000000002,"15min":684.42700000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.2849999999999999,"15min":1.1699999999999999},"min":{"1min":0.58899999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.306,"5min":6.1360000000000001,"15min":6.1360000000000001},"last":0.755}]},{"osd":2,"last update":"Fri Jan 29 13:56:22 2021","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.1919999999999999,"15min":1.242},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.492,"5min":7.2329999999999997,"15min":7.2329999999999997},"last":1.133},{"interface":"front","average":{"1min":36.773000000000003,"5min":8.3480000000000008,"15min":3.6339999999999999},"min":{"1min":0.68799999999999994,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":607.71000000000004,"5min":607.71000000000004,"15min":607.71000000000004},"last":0.67700000000000005}]},{"osd":3,"last update":"Fri Jan 29 13:56:22 2021","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":1.345,"15min":1.2470000000000001},"min":{"1min":0.66800000000000004,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.1779999999999999,"5min":6.048,"15min":6.048},"last":200.50399999999999},{"interface":"front","average":{"1min":29.335999999999999,"5min":6.8609999999999998,"15min":3.0449999999999999},"min":{"1min":0.83699999999999997,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":480.87299999999999,"5min":480.87299999999999,"15min":480.87299999999999},"last":1.21}]},{"osd":4,"last update":"Fri Jan 29 13:56:12 2021","interfaces":[{"interface":"back","average":{"1min":17.359999999999999,"5min":4.6619999999999999,"15min":2.456},"min":{"1min":0.629,"5min":0.62,"15min":0.62},"max":{"1min":263.584,"5min":263.584,"15min":263.584},"last":1.173},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.3069999999999999,"15min":1.417},"min":{"1min":0.78900000000000003,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.474,"5min":7.3449999999999998,"15min":7.3449999999999998},"last":0.82699999999999996}]},{"osd":5,"last update":"Fri Jan 29 13:56:22 2021","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.2969999999999999,"15min":1.349},"min":{"1min":0.86499999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.496,"5min":4.0069999999999997,"15min":4.0069999999999997},"last":1.034},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.6100000000000001,"15min":1.506},"min":{"1min":0.96499999999999997,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":1.3759999999999999,"5min":7.3140000000000001,"15min":7.3140000000000001},"last":1.052}]},{"osd":7,"last update":"Fri Jan 29 13:56:22 2021","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.4159999999999999,"15min":1.3380000000000001},"min":{"1min":0.90100000000000002,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.655,"5min":10.795,"15min":10.795},"last":1.375},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.3879999999999999,"15min":1.3919999999999999},"min":{"1min":0.85899999999999999,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.609,"5min":4.0579999999999998,"15min":4.0579999999999998},"last":0.99299999999999999}]}]}]}} , 2021-01-29T13:57:11.433 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-29T13:57:11.716 INFO:teuthology.orchestra.run.smithi183.stdout: 2021-01-30T06:14:57.768 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T13:57:44.175 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":301,"stamp":"2021-01-29T13:57:42.675417+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20305121604,"num_objects":6822,"num_object_clones":0,"num_object_copies":19368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":632,"num_objects_unfound":0,"num_objects_dirty":4824,"num_whiteouts":0,"num_read":9651,"num_read_kb":4830,"num_write":25509,"num_write_kb":14727220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3207,"num_bytes_recovered":7385874976,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":21,"num_bytes_hit_set_archive":12196,"num_flush":2862,"num_flush_kb":8348668,"num_evict":884,"num_evict_kb":2571590,"num_promote":4835,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8249,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15129,"ondisk_log_size":15129,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":45,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":25551644,"kb_used_data":25207644,"kb_used_omap":1,"kb_used_meta":343934,"kb_avail":729423076,"statfs":{"total":773094113280,"available":746929229824,"internally_reserved":0,"allocated":25812627456,"data_stored":30826570098,"data_compressed":523387886,"data_compressed_allocated":5107855360,"data_compressed_original":10215710720,"omap_allocated":1944,"internal_metadata":352188520},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-252,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":339084355,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.225468"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":"156","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.458800+0000","last_change":"2021-01-29T13:57:34.166076+0000","last_active":"2021-01-29T13:57:37.458800+0000","last_peered":"2021-01-29T13:57:37.458800+0000","last_clean":"2021-01-29T13:57:37.458800+0000","last_became_active":"2021-01-29T13:52:24.188931+0000","last_became_peered":"2021-01-29T13:52:24.188931+0000","last_unstale":"2021-01-29T13:57:37.458800+0000","last_undegraded":"2021-01-29T13:57:37.458800+0000","last_fullsized":"2021-01-29T13:57:37.458800+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:55:57.162808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:55:57.162808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"120'6449","reported_seq":"8600","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.461338+0000","last_change":"2021-01-29T13:55:45.625490+0000","last_active":"2021-01-29T13:57:37.461338+0000","last_peered":"2021-01-29T13:57:37.461338+0000","last_clean":"2021-01-29T13:57:37.461338+0000","last_became_active":"2021-01-29T13:52:39.830011+0000","last_became_peered":"2021-01-29T13:52:39.830011+0000","last_unstale":"2021-01-29T13:57:37.461338+0000","last_undegraded":"2021-01-29T13:57:37.461338+0000","last_fullsized":"2021-01-29T13:57:37.461338+0000","mapping_epoch":26,"log_start":"72'3403","ondisk_log_start":"72'3403","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'6250","last_scrub_stamp":"2021-01-29T13:55:45.625379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:38.811171+0000","last_clean_scrub_stamp":"2021-01-29T13:55:45.625379+0000","log_size":3046,"ondisk_log_size":3046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2928382292,"num_objects":986,"num_object_clones":0,"num_object_copies":1972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":2501,"num_read_kb":1252,"num_write":5879,"num_write_kb":1639641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":3315,"num_flush":761,"num_flush_kb":2192275,"num_evict":269,"num_evict_kb":760560,"num_promote":1249,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"117","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.497927+0000","last_change":"2021-01-29T13:55:42.661441+0000","last_active":"2021-01-29T13:57:37.497927+0000","last_peered":"2021-01-29T13:57:37.497927+0000","last_clean":"2021-01-29T13:57:37.497927+0000","last_became_active":"2021-01-29T13:53:29.493255+0000","last_became_peered":"2021-01-29T13:53:29.493255+0000","last_unstale":"2021-01-29T13:57:37.497927+0000","last_undegraded":"2021-01-29T13:57:37.497927+0000","last_fullsized":"2021-01-29T13:57:37.497927+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:55:42.661371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:55:42.661371+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3","version":"120'671","reported_seq":"1393","reported_epoch":"147","state":"active+remapped+backfilling","last_fresh":"2021-01-29T13:57:40.662249+0000","last_change":"2021-01-29T13:56:47.680232+0000","last_active":"2021-01-29T13:57:40.662249+0000","last_peered":"2021-01-29T13:57:40.662249+0000","last_clean":"2021-01-29T13:56:36.386610+0000","last_became_active":"2021-01-29T13:56:47.674484+0000","last_became_peered":"2021-01-29T13:56:47.674484+0000","last_unstale":"2021-01-29T13:57:40.662249+0000","last_undegraded":"2021-01-29T13:57:40.662249+0000","last_fullsized":"2021-01-29T13:57:40.662249+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'570","last_scrub_stamp":"2021-01-29T13:55:44.911108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:35.792587+0000","last_clean_scrub_stamp":"2021-01-29T13:55:44.911108+0000","log_size":671,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2038945493,"num_objects":671,"num_object_clones":0,"num_object_copies":2684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":404,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":671,"num_write_kb":1991482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":787,"num_bytes_recovered":2413500764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4,7],"acting":[6,2,4,7],"avail_no_missing":["6(0)","2(1)","4(2)","7(3)"],"object_location_counts":[{"shards":"2(1),4(2),6(0),7(3)","objects":671}],"blocked_by":[],"up_primary":0,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"147","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.457918+0000","last_change":"2021-01-29T13:57:01.550191+0000","last_active":"2021-01-29T13:57:37.457918+0000","last_peered":"2021-01-29T13:57:37.457918+0000","last_clean":"2021-01-29T13:57:37.457918+0000","last_became_active":"2021-01-29T13:52:23.765595+0000","last_became_peered":"2021-01-29T13:52:23.765595+0000","last_unstale":"2021-01-29T13:57:37.457918+0000","last_undegraded":"2021-01-29T13:57:37.457918+0000","last_fullsized":"2021-01-29T13:57:37.457918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:57:01.550087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:57:01.550087+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"117'710","reported_seq":"887","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.458693+0000","last_change":"2021-01-29T13:57:37.153012+0000","last_active":"2021-01-29T13:57:37.458693+0000","last_peered":"2021-01-29T13:57:37.458693+0000","last_clean":"2021-01-29T13:57:37.458693+0000","last_became_active":"2021-01-29T13:53:09.969063+0000","last_became_peered":"2021-01-29T13:53:09.969063+0000","last_unstale":"2021-01-29T13:57:37.458693+0000","last_undegraded":"2021-01-29T13:57:37.458693+0000","last_fullsized":"2021-01-29T13:57:37.458693+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'710","last_scrub_stamp":"2021-01-29T13:56:09.327903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:35.792587+0000","last_clean_scrub_stamp":"2021-01-29T13:56:09.327903+0000","log_size":710,"ondisk_log_size":710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2107977089,"num_objects":710,"num_object_clones":0,"num_object_copies":2840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":710,"num_write_kb":2058922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2,4],"acting":[5,7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"159","reported_epoch":"145","state":"active+clean","last_fresh":"2021-01-29T13:57:36.794315+0000","last_change":"2021-01-29T13:56:59.467725+0000","last_active":"2021-01-29T13:57:36.794315+0000","last_peered":"2021-01-29T13:57:36.794315+0000","last_clean":"2021-01-29T13:57:36.794315+0000","last_became_active":"2021-01-29T13:52:24.188578+0000","last_became_peered":"2021-01-29T13:52:24.188578+0000","last_unstale":"2021-01-29T13:57:36.794315+0000","last_undegraded":"2021-01-29T13:57:36.794315+0000","last_fullsized":"2021-01-29T13:57:36.794315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:56:59.467654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:56:59.467654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"142'4","reported_seq":"157","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.458197+0000","last_change":"2021-01-29T13:57:01.982145+0000","last_active":"2021-01-29T13:57:37.458197+0000","last_peered":"2021-01-29T13:57:37.458197+0000","last_clean":"2021-01-29T13:57:37.458197+0000","last_became_active":"2021-01-29T13:52:20.655015+0000","last_became_peered":"2021-01-29T13:52:20.655015+0000","last_unstale":"2021-01-29T13:57:37.458197+0000","last_undegraded":"2021-01-29T13:57:37.458197+0000","last_fullsized":"2021-01-29T13:57:37.458197+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'4","last_scrub_stamp":"2021-01-29T13:57:01.982073+0000","last_deep_scrub":"142'4","last_deep_scrub_stamp":"2021-01-29T13:57:01.982073+0000","last_clean_scrub_stamp":"2021-01-29T13:57:01.982073+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8249,"num_omap_keys":2,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"120'761","reported_seq":"3071","reported_epoch":"147","state":"active+remapped+backfilling","last_fresh":"2021-01-29T13:57:41.858978+0000","last_change":"2021-01-29T13:56:48.363777+0000","last_active":"2021-01-29T13:57:41.858978+0000","last_peered":"2021-01-29T13:57:41.858978+0000","last_clean":"2021-01-29T13:56:36.386422+0000","last_became_active":"2021-01-29T13:56:48.362095+0000","last_became_peered":"2021-01-29T13:56:48.362095+0000","last_unstale":"2021-01-29T13:57:41.858978+0000","last_undegraded":"2021-01-29T13:57:41.858978+0000","last_fullsized":"2021-01-29T13:57:41.858978+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'761","last_scrub_stamp":"2021-01-29T13:55:53.981049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:35.792587+0000","last_clean_scrub_stamp":"2021-01-29T13:55:53.981049+0000","log_size":761,"ondisk_log_size":761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2244508217,"num_objects":761,"num_object_clones":0,"num_object_copies":3044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":228,"num_objects_unfound":0,"num_objects_dirty":761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":761,"num_write_kb":2192275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1053,"num_bytes_recovered":3106526858,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7,0],"acting":[4,2,7,1],"avail_no_missing":["4(0)","1(3)","2(1)","7(2)"],"object_location_counts":[{"shards":"1(3),2(1),4(0),7(2)","objects":761}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"148","reported_epoch":"145","state":"active+clean","last_fresh":"2021-01-29T13:57:36.794261+0000","last_change":"2021-01-29T13:56:41.371116+0000","last_active":"2021-01-29T13:57:36.794261+0000","last_peered":"2021-01-29T13:57:36.794261+0000","last_clean":"2021-01-29T13:57:36.794261+0000","last_became_active":"2021-01-29T13:52:23.765671+0000","last_became_peered":"2021-01-29T13:52:23.765671+0000","last_unstale":"2021-01-29T13:57:36.794261+0000","last_undegraded":"2021-01-29T13:57:36.794261+0000","last_fullsized":"2021-01-29T13:57:36.794261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:56:41.371025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:56:41.371025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"118'720","reported_seq":"891","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.461342+0000","last_change":"2021-01-29T13:55:59.202455+0000","last_active":"2021-01-29T13:57:37.461342+0000","last_peered":"2021-01-29T13:57:37.461342+0000","last_clean":"2021-01-29T13:57:37.461342+0000","last_became_active":"2021-01-29T13:53:09.587252+0000","last_became_peered":"2021-01-29T13:53:09.587252+0000","last_unstale":"2021-01-29T13:57:37.461342+0000","last_undegraded":"2021-01-29T13:57:37.461342+0000","last_fullsized":"2021-01-29T13:57:37.461342+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'720","last_scrub_stamp":"2021-01-29T13:55:59.202349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:35.792587+0000","last_clean_scrub_stamp":"2021-01-29T13:55:59.202349+0000","log_size":720,"ondisk_log_size":720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2156154079,"num_objects":720,"num_object_clones":0,"num_object_copies":2880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":720,"num_write_kb":2105989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4,3],"acting":[6,2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"120'5986","reported_seq":"10744","reported_epoch":"147","state":"active+clean","last_fresh":"2021-01-29T13:57:38.478514+0000","last_change":"2021-01-29T13:57:38.478514+0000","last_active":"2021-01-29T13:57:38.478514+0000","last_peered":"2021-01-29T13:57:38.478514+0000","last_clean":"2021-01-29T13:57:38.478514+0000","last_became_active":"2021-01-29T13:57:38.475245+0000","last_became_peered":"2021-01-29T13:57:38.475245+0000","last_unstale":"2021-01-29T13:57:38.478514+0000","last_undegraded":"2021-01-29T13:57:38.478514+0000","last_fullsized":"2021-01-29T13:57:38.478514+0000","mapping_epoch":146,"log_start":"71'2904","ondisk_log_start":"71'2904","created":26,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'5955","last_scrub_stamp":"2021-01-29T13:55:51.275882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:38.811171+0000","last_clean_scrub_stamp":"2021-01-29T13:55:51.275882+0000","log_size":3082,"ondisk_log_size":3082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2947693900,"num_objects":985,"num_object_clones":0,"num_object_copies":1970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":2316,"num_read_kb":1158,"num_write":5435,"num_write_kb":1544566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1367,"num_bytes_recovered":1865847354,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":2771,"num_flush":671,"num_flush_kb":1991482,"num_evict":178,"num_evict_kb":526959,"num_promote":1172,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"171","reported_epoch":"147","state":"active+clean","last_fresh":"2021-01-29T13:57:39.387290+0000","last_change":"2021-01-29T13:57:39.387896+0000","last_active":"2021-01-29T13:57:39.387290+0000","last_peered":"2021-01-29T13:57:39.387290+0000","last_clean":"2021-01-29T13:57:39.387290+0000","last_became_active":"2021-01-29T13:53:09.377717+0000","last_became_peered":"2021-01-29T13:53:09.377717+0000","last_unstale":"2021-01-29T13:57:39.387290+0000","last_undegraded":"2021-01-29T13:57:39.387290+0000","last_fullsized":"2021-01-29T13:57:39.387290+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:55:33.131622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:55:33.131622+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"145","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.458896+0000","last_change":"2021-01-29T13:56:53.807344+0000","last_active":"2021-01-29T13:57:37.458896+0000","last_peered":"2021-01-29T13:57:37.458896+0000","last_clean":"2021-01-29T13:57:37.458896+0000","last_became_active":"2021-01-29T13:52:40.828334+0000","last_became_peered":"2021-01-29T13:52:40.828334+0000","last_unstale":"2021-01-29T13:57:37.458896+0000","last_undegraded":"2021-01-29T13:57:37.458896+0000","last_fullsized":"2021-01-29T13:57:37.458896+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:56:53.807251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:56:53.807251+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"117'6273","reported_seq":"8324","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.497913+0000","last_change":"2021-01-29T13:55:55.445357+0000","last_active":"2021-01-29T13:57:37.497913+0000","last_peered":"2021-01-29T13:57:37.497913+0000","last_clean":"2021-01-29T13:57:37.497913+0000","last_became_active":"2021-01-29T13:53:09.369472+0000","last_became_peered":"2021-01-29T13:53:09.369472+0000","last_unstale":"2021-01-29T13:57:37.497913+0000","last_undegraded":"2021-01-29T13:57:37.497913+0000","last_fullsized":"2021-01-29T13:57:37.497913+0000","mapping_epoch":42,"log_start":"71'3203","ondisk_log_start":"71'3203","created":26,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'6273","last_scrub_stamp":"2021-01-29T13:55:55.445229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:38.811171+0000","last_clean_scrub_stamp":"2021-01-29T13:55:55.445229+0000","log_size":3070,"ondisk_log_size":3070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2935339657,"num_objects":992,"num_object_clones":0,"num_object_copies":1984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":496,"num_whiteouts":0,"num_read":2414,"num_read_kb":1208,"num_write":5647,"num_write_kb":1593121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":3044,"num_flush":710,"num_flush_kb":2058922,"num_evict":219,"num_evict_kb":640347,"num_promote":1206,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"29","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.463934+0000","last_change":"2021-01-29T13:56:46.442031+0000","last_active":"2021-01-29T13:57:37.463934+0000","last_peered":"2021-01-29T13:57:37.463934+0000","last_clean":"2021-01-29T13:57:37.463934+0000","last_became_active":"2021-01-29T13:56:46.441589+0000","last_became_peered":"2021-01-29T13:56:46.441589+0000","last_unstale":"2021-01-29T13:57:37.463934+0000","last_undegraded":"2021-01-29T13:57:37.463934+0000","last_fullsized":"2021-01-29T13:57:37.463934+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:55:55.134762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:55:55.134762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"118'6270","reported_seq":"8356","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.464735+0000","last_change":"2021-01-29T13:57:04.235378+0000","last_active":"2021-01-29T13:57:37.464735+0000","last_peered":"2021-01-29T13:57:37.464735+0000","last_clean":"2021-01-29T13:57:37.464735+0000","last_became_active":"2021-01-29T13:52:39.823642+0000","last_became_peered":"2021-01-29T13:52:39.823642+0000","last_unstale":"2021-01-29T13:57:37.464735+0000","last_undegraded":"2021-01-29T13:57:37.464735+0000","last_fullsized":"2021-01-29T13:57:37.464735+0000","mapping_epoch":26,"log_start":"71'3205","ondisk_log_start":"71'3205","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6270","last_scrub_stamp":"2021-01-29T13:57:04.235268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:38.811171+0000","last_clean_scrub_stamp":"2021-01-29T13:57:04.235268+0000","log_size":3065,"ondisk_log_size":3065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2946120877,"num_objects":995,"num_object_clones":0,"num_object_copies":1990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":2420,"num_read_kb":1212,"num_write":5684,"num_write_kb":1601214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":3066,"num_flush":720,"num_flush_kb":2105989,"num_evict":218,"num_evict_kb":643724,"num_promote":1208,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":4,"stat_sum":{"num_bytes":11757536726,"num_objects":3958,"num_object_clones":0,"num_object_copies":7916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1960,"num_whiteouts":0,"num_read":9651,"num_read_kb":4830,"num_write":22645,"num_write_kb":6378542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1367,"num_bytes_recovered":1865847354,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":21,"num_bytes_hit_set_archive":12196,"num_flush":2862,"num_flush_kb":8348668,"num_evict":884,"num_evict_kb":2571590,"num_promote":4835,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11983056896,"data_stored":11892361364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12263,"ondisk_log_size":12263,"up":8,"acting":8,"num_store_stats":6},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":8547584878,"num_objects":2862,"num_object_clones":0,"num_object_copies":11448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":632,"num_objects_unfound":0,"num_objects_dirty":2862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2862,"num_write_kb":8348668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1840,"num_bytes_recovered":5520027622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13793206272,"data_stored":18905419776,"data_compressed":523841489,"data_compressed_allocated":5112213504,"data_compressed_original":10224427008,"omap_allocated":0,"internal_metadata":0},"log_size":2862,"ondisk_log_size":2862,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8249,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":0,"up_from":124,"seq":532575944719,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2622556,"kb_used_data":2579220,"kb_used_omap":0,"kb_used_meta":43327,"kb_avail":91749284,"statfs":{"total":96636764160,"available":93951266816,"internally_reserved":0,"allocated":2641121280,"data_stored":3110022318,"data_compressed":49186610,"data_compressed_allocated":479526912,"data_compressed_original":959053824,"omap_allocated":972,"internal_metadata":44366900},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":1.0149999999999999}]},{"osd":2,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":2.3559999999999999,"5min":2.3559999999999999,"15min":2.3559999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.93300000000000005}]},{"osd":3,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":7.8730000000000002,"15min":7.8730000000000002},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":170.62200000000001,"5min":170.62200000000001,"15min":170.62200000000001},"last":170.62200000000001},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":2.3149999999999999,"5min":2.3149999999999999,"15min":2.3149999999999999},"last":1.258}]},{"osd":4,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.117,"15min":1.117},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":1.056},{"interface":"front","average":{"1min":1.141,"5min":1.141,"15min":1.141},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.1059999999999999,"5min":2.1059999999999999,"15min":2.1059999999999999},"last":1.2370000000000001}]},{"osd":5,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":6.9889999999999999,"5min":6.9889999999999999,"15min":6.9889999999999999},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":136.00800000000001,"5min":136.00800000000001,"15min":136.00800000000001},"last":1.294},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":0.96299999999999997}]},{"osd":6,"last update":"Fri Jan 29 13:57:37 2021","interfaces":[{"interface":"back","average":{"1min":22.164000000000001,"5min":22.164000000000001,"15min":22.164000000000001},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":484.286,"5min":484.286,"15min":484.286},"last":1.1140000000000001},{"interface":"front","average":{"1min":24.145,"5min":24.145,"15min":24.145},"min":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"max":{"1min":529.46000000000004,"5min":529.46000000000004,"15min":529.46000000000004},"last":529.46000000000004}]},{"osd":7,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.274,"15min":1.274},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"max":{"1min":2.3879999999999999,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.264,"5min":1.264,"15min":1.264},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.349}]}]},{"osd":7,"up_from":15,"seq":64424509508,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2311472,"kb_used_data":2281064,"kb_used_omap":0,"kb_used_meta":30400,"kb_avail":92060368,"statfs":{"total":96636764160,"available":94269816832,"internally_reserved":0,"allocated":2335809536,"data_stored":3200494326,"data_compressed":88587048,"data_compressed_allocated":865042432,"data_compressed_original":1730084864,"omap_allocated":0,"internal_metadata":31129600},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.70299999999999996},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":1.004}]},{"osd":1,"last update":"Fri Jan 29 13:57:30 2021","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.242,"15min":1.46},"min":{"1min":0.64900000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.671,"5min":11.195,"15min":11.195},"last":0.91200000000000003},{"interface":"front","average":{"1min":50.901000000000003,"5min":26.797999999999998,"15min":33.463000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":951.06700000000001,"5min":951.06700000000001,"15min":951.06700000000001},"last":0.79000000000000004}]},{"osd":2,"last update":"Fri Jan 29 13:57:30 2021","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":10.314,"15min":4.5030000000000001},"min":{"1min":0.72099999999999997,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.147,"5min":512.423,"15min":512.423},"last":0.73799999999999999},{"interface":"front","average":{"1min":27.712,"5min":7.3419999999999996,"15min":3.5150000000000001},"min":{"1min":0.68600000000000005,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":505.935,"5min":505.935,"15min":505.935},"last":0.95299999999999996}]},{"osd":3,"last update":"Fri Jan 29 13:57:30 2021","interfaces":[{"interface":"back","average":{"1min":52.369999999999997,"5min":13.276,"15min":5.1639999999999997},"min":{"1min":0.749,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":977.31200000000001,"5min":977.31200000000001,"15min":977.31200000000001},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.381,"15min":1.581},"min":{"1min":1.006,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.7789999999999999,"5min":11.146000000000001,"15min":11.146000000000001},"last":1.3049999999999999}]},{"osd":4,"last update":"Fri Jan 29 13:57:17 2021","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.254,"15min":1.3340000000000001},"min":{"1min":0.79900000000000004,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.7450000000000001,"5min":5.8449999999999998,"15min":5.8449999999999998},"last":1.0469999999999999},{"interface":"front","average":{"1min":16.503,"5min":13.058,"15min":19.832000000000001},"min":{"1min":0.79700000000000004,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":309.072,"5min":391.66699999999997,"15min":391.66699999999997},"last":0.98199999999999998}]},{"osd":5,"last update":"Fri Jan 29 13:57:30 2021","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":3.3260000000000001,"15min":2.242},"min":{"1min":0.76700000000000002,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.625,"5min":175.83199999999999,"15min":175.83199999999999},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.294,"15min":1.2130000000000001},"min":{"1min":0.97099999999999997,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.8120000000000001,"5min":4.4020000000000001,"15min":4.4020000000000001},"last":1.3260000000000001}]},{"osd":6,"last update":"Fri Jan 29 13:57:30 2021","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.552,"15min":1.532},"min":{"1min":1.119,"5min":0.754,"15min":0.754},"max":{"1min":1.881,"5min":10.063000000000001,"15min":10.063000000000001},"last":1.345},{"interface":"front","average":{"1min":8.1750000000000007,"5min":9.5399999999999991,"15min":26.190000000000001},"min":{"1min":0.95299999999999996,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":132.83500000000001,"5min":623.01999999999998,"15min":623.01999999999998},"last":1.278}]}]},{"osd":2,"up_from":15,"seq":64424509507,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5660992,"kb_used_data":5606840,"kb_used_omap":0,"kb_used_meta":54144,"kb_avail":88710848,"statfs":{"total":96636764160,"available":90839908352,"internally_reserved":0,"allocated":5741404160,"data_stored":6879489840,"data_compressed":118712623,"data_compressed_allocated":1158184960,"data_compressed_original":2316369920,"omap_allocated":0,"internal_metadata":55443456},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.94299999999999995}]},{"osd":1,"last update":"Fri Jan 29 13:57:29 2021","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":17.460000000000001,"15min":6.4560000000000004},"min":{"1min":0.63200000000000001,"5min":0.499,"15min":0.499},"max":{"1min":1.3999999999999999,"5min":721.11000000000001,"15min":721.11000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.95499999999999996,"5min":21.443999999999999,"15min":7.8150000000000004},"min":{"1min":0.71899999999999997,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.3460000000000001,"5min":1000.376,"15min":1000.376},"last":0.96999999999999997}]},{"osd":3,"last update":"Fri Jan 29 13:57:29 2021","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":11.378,"15min":4.343},"min":{"1min":0.60299999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1160000000000001,"5min":501.61200000000002,"15min":501.61200000000002},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.96799999999999997,"5min":1.1819999999999999,"15min":1.0620000000000001},"min":{"1min":0.65900000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.3500000000000001,"5min":6.1870000000000003,"15min":6.1870000000000003},"last":0.90100000000000002}]},{"osd":4,"last update":"Fri Jan 29 13:57:18 2021","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.4099999999999999,"15min":1.5069999999999999},"min":{"1min":0.72699999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.6160000000000001,"5min":3.694,"15min":3.694},"last":1.1850000000000001},{"interface":"front","average":{"1min":77.519000000000005,"5min":16.635999999999999,"15min":6.4619999999999997},"min":{"1min":0.96199999999999997,"5min":0.871,"15min":0.871},"max":{"1min":874.88400000000001,"5min":874.88400000000001,"15min":874.88400000000001},"last":1.2050000000000001}]},{"osd":5,"last update":"Fri Jan 29 13:57:29 2021","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":5.7699999999999996,"15min":2.762},"min":{"1min":0.79800000000000004,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.349,"5min":183.761,"15min":183.761},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.1579999999999999,"5min":9.484,"15min":4.0389999999999997},"min":{"1min":0.88400000000000001,"5min":0.746,"15min":0.746},"max":{"1min":1.5469999999999999,"5min":521.02499999999998,"15min":521.02499999999998},"last":0.99099999999999999}]},{"osd":6,"last update":"Fri Jan 29 13:57:29 2021","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.359,"15min":1.2969999999999999},"min":{"1min":0.93500000000000005,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.544,"5min":3.46,"15min":3.46},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.135,"5min":1.4319999999999999,"15min":1.276},"min":{"1min":0.87,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.4590000000000001,"5min":7.0209999999999999,"15min":7.0209999999999999},"last":0.88500000000000001}]},{"osd":7,"last update":"Fri Jan 29 13:57:29 2021","interfaces":[{"interface":"back","average":{"1min":22.652000000000001,"5min":12.759,"15min":5.1719999999999997},"min":{"1min":0.98399999999999999,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":427.892,"5min":591.18899999999996,"15min":591.18899999999996},"last":1.226},{"interface":"front","average":{"1min":1.26,"5min":11.763999999999999,"15min":4.8159999999999998},"min":{"1min":0.95499999999999996,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.738,"5min":923.40099999999995,"15min":923.40099999999995},"last":1.103}]}]},{"osd":1,"up_from":15,"seq":64424509508,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2154968,"kb_used_data":2115152,"kb_used_omap":0,"kb_used_meta":39808,"kb_avail":92216872,"statfs":{"total":96636764160,"available":94430076928,"internally_reserved":0,"allocated":2165915648,"data_stored":2457997850,"data_compressed":31021897,"data_compressed_allocated":302632960,"data_compressed_original":605265920,"omap_allocated":0,"internal_metadata":40763392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:54:25 2021","interfaces":[{"interface":"back","average":{"1min":1.7629999999999999,"5min":1.0349999999999999,"15min":0.91300000000000003},"min":{"1min":0.63300000000000001,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":5.0629999999999997,"5min":5.0629999999999997,"15min":5.0629999999999997},"last":0.70799999999999996},{"interface":"front","average":{"1min":39.090000000000003,"5min":8.5060000000000002,"15min":3.4079999999999999},"min":{"1min":0.58699999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":752.73400000000004,"5min":752.73400000000004,"15min":752.73400000000004},"last":0.65200000000000002}]},{"osd":2,"last update":"Fri Jan 29 13:57:29 2021","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":16.138999999999999,"15min":27.606000000000002},"min":{"1min":0.57199999999999995,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.153,"5min":832.21600000000001,"15min":832.21600000000001},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.85299999999999998,"5min":11.237,"15min":4.4100000000000001},"min":{"1min":0.59099999999999997,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.1870000000000001,"5min":899.43399999999997,"15min":899.43399999999997},"last":0.84199999999999997}]},{"osd":3,"last update":"Fri Jan 29 13:57:29 2021","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":1.296,"15min":1.0960000000000001},"min":{"1min":0.63300000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.3169999999999999,"5min":6.0579999999999998,"15min":6.0579999999999998},"last":0.876},{"interface":"front","average":{"1min":0.84299999999999997,"5min":3.3029999999999999,"15min":9.1440000000000001},"min":{"1min":0.55100000000000005,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.1220000000000001,"5min":244.60900000000001,"15min":244.60900000000001},"last":0.82499999999999996}]},{"osd":4,"last update":"Fri Jan 29 13:57:18 2021","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.325,"15min":1.5880000000000001},"min":{"1min":0.71399999999999997,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.254,"5min":5.242,"15min":5.242},"last":0.95099999999999996},{"interface":"front","average":{"1min":1.079,"5min":1.6839999999999999,"15min":2.0870000000000002},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.2310000000000001,"5min":6.2590000000000003,"15min":6.2590000000000003},"last":1.165}]},{"osd":5,"last update":"Fri Jan 29 13:57:29 2021","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":14.631,"15min":5.681},"min":{"1min":0.68300000000000005,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.3440000000000001,"5min":701.73400000000004,"15min":701.73400000000004},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.302,"15min":1.248},"min":{"1min":0.72899999999999998,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.359,"5min":5.758,"15min":5.758},"last":1.0940000000000001}]},{"osd":6,"last update":"Fri Jan 29 13:57:30 2021","interfaces":[{"interface":"back","average":{"1min":48.951999999999998,"5min":10.974,"15min":4.5209999999999999},"min":{"1min":0.89800000000000002,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":862.01999999999998,"5min":862.01999999999998,"15min":862.01999999999998},"last":1.046},{"interface":"front","average":{"1min":1.1659999999999999,"5min":2.7810000000000001,"15min":5.8049999999999997},"min":{"1min":0.83799999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.4790000000000001,"5min":132.35300000000001,"15min":132.35300000000001},"last":1.1459999999999999}]},{"osd":7,"last update":"Fri Jan 29 13:57:29 2021","interfaces":[{"interface":"back","average":{"1min":45.155999999999999,"5min":17.600000000000001,"15min":6.7450000000000001},"min":{"1min":0.83999999999999997,"5min":0.752,"15min":0.752},"max":{"1min":749.81700000000001,"5min":749.81700000000001,"15min":749.81700000000001},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.4530000000000001,"15min":1.3979999999999999},"min":{"1min":0.85199999999999998,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":1.4790000000000001,"5min":5.9450000000000003,"15min":5.9450000000000003},"last":1.216}]}]},{"osd":3,"up_from":15,"seq":64424509508,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3443368,"kb_used_data":3391712,"kb_used_omap":0,"kb_used_meta":51648,"kb_avail":90928472,"statfs":{"total":96636764160,"available":93110755328,"internally_reserved":0,"allocated":3473113088,"data_stored":3742649427,"data_compressed":29735076,"data_compressed_allocated":290086912,"data_compressed_original":580173824,"omap_allocated":0,"internal_metadata":52887552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":29.826000000000001,"5min":29.826000000000001,"15min":29.826000000000001},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":301.69099999999997,"5min":301.69099999999997,"15min":301.69099999999997},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.613,"5min":1.613,"15min":1.613},"last":0.66800000000000004}]},{"osd":1,"last update":"Fri Jan 29 13:57:31 2021","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.96599999999999997,"15min":0.91000000000000003},"min":{"1min":0.57699999999999996,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.1339999999999999,"5min":2.5819999999999999,"15min":2.5819999999999999},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.86499999999999999,"5min":10.300000000000001,"15min":4.0890000000000004},"min":{"1min":0.59899999999999998,"5min":0.496,"15min":0.496},"max":{"1min":1.0600000000000001,"5min":977.78200000000004,"15min":977.78200000000004},"last":0.88500000000000001}]},{"osd":2,"last update":"Fri Jan 29 13:57:31 2021","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.181,"15min":1.079},"min":{"1min":0.65700000000000003,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.3919999999999999,"5min":5.7539999999999996,"15min":5.7539999999999996},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.3009999999999999,"15min":1.407},"min":{"1min":0.82999999999999996,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.2190000000000001,"5min":6.4509999999999996,"15min":6.4509999999999996},"last":1.1819999999999999}]},{"osd":4,"last update":"Fri Jan 29 13:57:15 2021","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":11.065,"15min":4.5490000000000004},"min":{"1min":0.71099999999999997,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.4890000000000001,"5min":916.09500000000003,"15min":916.09500000000003},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.131,"5min":11.628,"15min":4.859},"min":{"1min":0.81899999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.46,"5min":977.91499999999996,"15min":977.91499999999996},"last":1.0169999999999999}]},{"osd":5,"last update":"Fri Jan 29 13:57:31 2021","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":6.8769999999999998,"15min":3.1320000000000001},"min":{"1min":0.88500000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.3460000000000001,"5min":503.93000000000001,"15min":503.93000000000001},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.0269999999999999,"5min":6.7670000000000003,"15min":3.0150000000000001},"min":{"1min":0.83999999999999997,"5min":0.751,"15min":0.751},"max":{"1min":1.2569999999999999,"5min":503.74299999999999,"15min":503.74299999999999},"last":0.96199999999999997}]},{"osd":6,"last update":"Fri Jan 29 13:57:31 2021","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.4279999999999999,"15min":1.3440000000000001},"min":{"1min":1.0249999999999999,"5min":0.751,"15min":0.751},"max":{"1min":1.4710000000000001,"5min":4.7539999999999996,"15min":4.7539999999999996},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.141,"5min":10.393000000000001,"15min":34.238},"min":{"1min":0.876,"5min":0.876,"15min":0.876},"max":{"1min":1.3899999999999999,"5min":851.89099999999996,"15min":851.89099999999996},"last":1.1419999999999999}]},{"osd":7,"last update":"Fri Jan 29 13:57:31 2021","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.514,"15min":1.4910000000000001},"min":{"1min":1.0109999999999999,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.629,"5min":6.4240000000000004,"15min":6.4240000000000004},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.6830000000000001,"15min":1.917},"min":{"1min":1.038,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.6639999999999999,"5min":15.597,"15min":15.597},"last":1.246}]}]},{"osd":4,"up_from":42,"seq":180388626490,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4411548,"kb_used_data":4364052,"kb_used_omap":0,"kb_used_meta":47487,"kb_avail":89960292,"statfs":{"total":96636764160,"available":92119339008,"internally_reserved":0,"allocated":4468789248,"data_stored":5613909741,"data_compressed":118370768,"data_compressed_allocated":1155686400,"data_compressed_original":2311372800,"omap_allocated":972,"internal_metadata":48626740},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":1.095}]},{"osd":1,"last update":"Fri Jan 29 13:57:19 2021","interfaces":[{"interface":"back","average":{"1min":9.0939999999999994,"5min":31.199000000000002,"15min":58.671999999999997},"min":{"1min":0.747,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":177.846,"5min":924.63300000000004,"15min":924.63300000000004},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.151,"5min":15.670999999999999,"15min":5.9130000000000003},"min":{"1min":0.79300000000000004,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.637,"5min":898.86900000000003,"15min":898.86900000000003},"last":1.4630000000000001}]},{"osd":2,"last update":"Fri Jan 29 13:57:19 2021","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":11.949,"15min":4.7149999999999999},"min":{"1min":0.71299999999999997,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.6020000000000001,"5min":857.78899999999999,"15min":857.78899999999999},"last":1.395},{"interface":"front","average":{"1min":0.95099999999999996,"5min":9.8019999999999996,"15min":4.0369999999999999},"min":{"1min":0.76300000000000001,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.198,"5min":779.38400000000001,"15min":779.38400000000001},"last":0.86399999999999999}]},{"osd":3,"last update":"Fri Jan 29 13:57:19 2021","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":9.625,"15min":13.391},"min":{"1min":0.69799999999999995,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.423,"5min":311.68299999999999,"15min":311.68299999999999},"last":1.032},{"interface":"front","average":{"1min":35.929000000000002,"5min":8.1660000000000004,"15min":3.5739999999999998},"min":{"1min":0.83999999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":765.98299999999995,"5min":765.98299999999995,"15min":765.98299999999995},"last":1.4319999999999999}]},{"osd":5,"last update":"Fri Jan 29 13:57:19 2021","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":3.7069999999999999,"15min":1.956},"min":{"1min":0.61899999999999999,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.238,"5min":239.99199999999999,"15min":239.99199999999999},"last":1.012},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.1639999999999999,"15min":1.1579999999999999},"min":{"1min":0.84499999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.464,"5min":3.1749999999999998,"15min":3.1749999999999998},"last":1.141}]},{"osd":6,"last update":"Fri Jan 29 13:57:19 2021","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.2370000000000001,"15min":1.2050000000000001},"min":{"1min":0.73599999999999999,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.657,"5min":5.5890000000000004,"15min":5.5890000000000004},"last":1.492},{"interface":"front","average":{"1min":1.113,"5min":1.2110000000000001,"15min":1.282},"min":{"1min":0.86599999999999999,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.6719999999999999,"5min":3.6880000000000002,"15min":3.6880000000000002},"last":1.1220000000000001}]},{"osd":7,"last update":"Fri Jan 29 13:57:19 2021","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.2869999999999999,"15min":1.2809999999999999},"min":{"1min":0.98299999999999998,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":1.5620000000000001,"5min":3.0169999999999999,"15min":3.0169999999999999},"last":1.236},{"interface":"front","average":{"1min":33.942,"5min":7.8579999999999997,"15min":3.5139999999999998},"min":{"1min":1.0109999999999999,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":719.03700000000003,"5min":719.03700000000003,"15min":719.03700000000003},"last":1.5209999999999999}]}]},{"osd":5,"up_from":15,"seq":64424509507,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":765928,"kb_used_data":751904,"kb_used_omap":0,"kb_used_meta":14016,"kb_avail":93605912,"statfs":{"total":96636764160,"available":95852453888,"internally_reserved":0,"allocated":769949696,"data_stored":1055829374,"data_compressed":29380114,"data_compressed_allocated":286253056,"data_compressed_original":572506112,"omap_allocated":0,"internal_metadata":14352384},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:57:39 2021","interfaces":[{"interface":"back","average":{"1min":24.538,"5min":24.538,"15min":24.538},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":474.77499999999998,"5min":474.77499999999998,"15min":474.77499999999998},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.98799999999999999}]},{"osd":1,"last update":"Fri Jan 29 13:57:21 2021","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.96799999999999997,"15min":0.87},"min":{"1min":0.58299999999999996,"5min":0.498,"15min":0.498},"max":{"1min":1.2350000000000001,"5min":3.1819999999999999,"15min":3.1819999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.1499999999999999,"5min":11.096,"15min":4.2889999999999997},"min":{"1min":0.84999999999999998,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.518,"5min":732.80700000000002,"15min":732.80700000000002},"last":0.91200000000000003}]},{"osd":2,"last update":"Fri Jan 29 13:57:21 2021","interfaces":[{"interface":"back","average":{"1min":12.396000000000001,"5min":10.401999999999999,"15min":4.2919999999999998},"min":{"1min":0.81599999999999995,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":225.11799999999999,"5min":649.82899999999995,"15min":649.82899999999995},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.125,"15min":1.056},"min":{"1min":0.81799999999999995,"5min":0.623,"15min":0.623},"max":{"1min":1.256,"5min":4.1799999999999997,"15min":4.1799999999999997},"last":1.2010000000000001}]},{"osd":3,"last update":"Fri Jan 29 13:57:21 2021","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":1.1160000000000001,"15min":1.1259999999999999},"min":{"1min":0.67700000000000005,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.4730000000000001,"5min":4.0890000000000004,"15min":4.0890000000000004},"last":0.86099999999999999},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.27,"15min":1.228},"min":{"1min":0.92300000000000004,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.7350000000000001,"5min":4.827,"15min":4.827},"last":0.95499999999999996}]},{"osd":4,"last update":"Fri Jan 29 13:57:18 2021","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":13.199,"15min":16.175999999999998},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.6000000000000001,"5min":514.74199999999996,"15min":514.74199999999996},"last":1.228},{"interface":"front","average":{"1min":39.335999999999999,"5min":8.8729999999999993,"15min":3.7469999999999999},"min":{"1min":0.83599999999999997,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":762.10699999999997,"5min":762.10699999999997,"15min":762.10699999999997},"last":1.1499999999999999}]},{"osd":6,"last update":"Fri Jan 29 13:57:21 2021","interfaces":[{"interface":"back","average":{"1min":37.015999999999998,"5min":16.678000000000001,"15min":6.4059999999999997},"min":{"1min":1.0900000000000001,"5min":0.622,"15min":0.622},"max":{"1min":714.803,"5min":714.803,"15min":714.803},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.4630000000000001,"15min":1.27},"min":{"1min":1.119,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.8320000000000001,"5min":4.8700000000000001,"15min":4.8700000000000001},"last":1.2669999999999999}]},{"osd":7,"last update":"Fri Jan 29 13:57:21 2021","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.423,"15min":1.5049999999999999},"min":{"1min":1.0409999999999999,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.851,"5min":9.0109999999999992,"15min":9.0109999999999992},"last":1.282},{"interface":"front","average":{"1min":1.419,"5min":1.6479999999999999,"15min":1.758},"min":{"1min":1.1319999999999999,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":1.96,"5min":9.593,"15min":9.593},"last":1.3340000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509507,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4180812,"kb_used_data":4117700,"kb_used_omap":0,"kb_used_meta":63104,"kb_avail":90191028,"statfs":{"total":96636764160,"available":92355612672,"internally_reserved":0,"allocated":4216524800,"data_stored":4766177222,"data_compressed":58393750,"data_compressed_allocated":570441728,"data_compressed_original":1140883456,"omap_allocated":0,"internal_metadata":64618496},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:57:35 2021","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.5740000000000001},"last":1.1299999999999999},{"interface":"front","average":{"1min":31.253,"5min":31.253,"15min":31.253},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":729.27700000000004,"5min":729.27700000000004,"15min":729.27700000000004},"last":0.96599999999999997}]},{"osd":1,"last update":"Fri Jan 29 13:57:26 2021","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":8.0850000000000009,"15min":26.884},"min":{"1min":0.626,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.5029999999999999,"5min":684.42700000000002,"15min":684.42700000000002},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.26,"15min":1.1619999999999999},"min":{"1min":0.69699999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.319,"5min":6.1360000000000001,"15min":6.1360000000000001},"last":1.157}]},{"osd":2,"last update":"Fri Jan 29 13:57:26 2021","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.167,"15min":1.234},"min":{"1min":0.91800000000000004,"5min":0.498,"15min":0.498},"max":{"1min":1.383,"5min":7.2329999999999997,"15min":7.2329999999999997},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.0349999999999999,"5min":8.3000000000000007,"15min":3.6179999999999999},"min":{"1min":0.60499999999999998,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.542,"5min":607.71000000000004,"15min":607.71000000000004},"last":0.86699999999999999}]},{"osd":3,"last update":"Fri Jan 29 13:57:26 2021","interfaces":[{"interface":"back","average":{"1min":29.053999999999998,"5min":6.9160000000000004,"15min":3.1040000000000001},"min":{"1min":0.77700000000000002,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":501.613,"5min":501.613,"15min":501.613},"last":1.325},{"interface":"front","average":{"1min":1.1919999999999999,"5min":6.8719999999999999,"15min":3.0489999999999999},"min":{"1min":0.88500000000000001,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.4199999999999999,"5min":480.87299999999999,"15min":480.87299999999999},"last":1.1819999999999999}]},{"osd":4,"last update":"Fri Jan 29 13:57:13 2021","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":4.6059999999999999,"15min":2.4380000000000002},"min":{"1min":0.71799999999999997,"5min":0.62,"15min":0.62},"max":{"1min":1.6439999999999999,"5min":263.584,"15min":263.584},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.238,"15min":1.3939999999999999},"min":{"1min":0.82699999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.6200000000000001,"5min":7.3449999999999998,"15min":7.3449999999999998},"last":1.423}]},{"osd":5,"last update":"Fri Jan 29 13:57:26 2021","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.27,"15min":1.3400000000000001},"min":{"1min":1.0049999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.6020000000000001,"5min":4.0069999999999997,"15min":4.0069999999999997},"last":1.27},{"interface":"front","average":{"1min":1.25,"5min":1.569,"15min":1.492},"min":{"1min":1.012,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":1.5409999999999999,"5min":7.3140000000000001,"15min":7.3140000000000001},"last":1.5940000000000001}]},{"osd":7,"last update":"Fri Jan 29 13:57:26 2021","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.4159999999999999,"15min":1.3380000000000001},"min":{"1min":0.91100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.581,"5min":10.795,"15min":10.795},"last":1.655},{"interface":"front","average":{"1min":1.278,"5min":1.365,"15min":1.3839999999999999},"min":{"1min":0.99299999999999999,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.722,"5min":4.0579999999999998,"15min":4.0579999999999998},"last":1.383}]}]}]}} , 2021-01-29T13:57:47.178 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-29T13:57:47.465 INFO:teuthology.orchestra.run.smithi183.stdout: 2021-01-30T06:14:57.773 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T13:58:10.671 INFO:teuthology.orchestra.run.smithi183.stdout:} , 2021-01-29T13:58:10.682 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 , instructed osd(s) 2 to deep-scrub 2021-01-30T06:14:57.774 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 179, in _raise_for_status , raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 10000 --objects 6600 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 1200 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2021-01-30T06:14:57.775 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T13:59:00.217 DEBUG:teuthology.orchestra.run.smithi183:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- , 2021-01-29T14:01:20.328 INFO:tasks.ceph:Archiving logs... , 2021-01-29T14:01:20.329 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/log/ceph to /home/teuthworker/archive/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840036/remote/smithi183/log 2021-01-30T06:14:57.776 WARNING:scrape:Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 368, in archive , yield 2021-01-30T06:14:58.002 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-297-g27dc1eb3 (27dc1eb32728beb0c69aa65131bc609ef680c0aa) quincy (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f30135d315a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x55fd1cf11fe6] 2021-01-30T06:14:58.004 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T13:55:49.136 INFO:teuthology.orchestra.run.smithi142.stdout:ERROR: (22) Invalid argument , 2021-01-29T13:55:49.136 INFO:teuthology.orchestra.run.smithi142.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. , 2021-01-29T13:55:49.151 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2021-01-30T06:14:58.004 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T13:56:38.516 INFO:teuthology.orchestra.run.smithi183.stdout:oload 1000 , 2021-01-29T13:56:38.516 INFO:teuthology.orchestra.run.smithi183.stdout:max_change 3 , 2021-01-29T13:56:38.517 INFO:teuthology.orchestra.run.smithi183.stdout:max_change_osds 4 2021-01-30T06:14:58.006 WARNING:scrape:Ignoring malparsed backtrace: set osd.3 primary-affinity to 1 (8655362) , 2021-01-29T13:56:51.495 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 , 2021-01-29T13:56:52.480+0000 7f8f78ab0700 -1 osd.2 132 handle_osd_map injecting map crc failure 2021-01-30T06:14:58.012 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T13:57:08.429 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":278,"stamp":"2021-01-29T13:57:06.667864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20305121604,"num_objects":6822,"num_object_clones":0,"num_object_copies":19368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2076,"num_objects_unfound":0,"num_objects_dirty":4824,"num_whiteouts":0,"num_read":9651,"num_read_kb":4830,"num_write":25509,"num_write_kb":14727220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2477,"num_bytes_recovered":5602499186,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":21,"num_bytes_hit_set_archive":12196,"num_flush":2862,"num_flush_kb":8348668,"num_evict":884,"num_evict_kb":2571590,"num_promote":4835,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8249,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15129,"ondisk_log_size":15129,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":45,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":24946968,"kb_used_data":24625140,"kb_used_omap":1,"kb_used_meta":321598,"kb_avail":730027752,"statfs":{"total":773094113280,"available":747548418048,"internally_reserved":0,"allocated":25216143360,"data_stored":30051268153,"data_compressed":504955484,"data_compressed_allocated":4927905792,"data_compressed_original":9855811584,"omap_allocated":1944,"internal_metadata":329316456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-170,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":394035071,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8249,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.133172"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":"145","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.847683+0000","last_change":"2021-01-29T13:55:57.162929+0000","last_active":"2021-01-29T13:57:00.847683+0000","last_peered":"2021-01-29T13:57:00.847683+0000","last_clean":"2021-01-29T13:57:00.847683+0000","last_became_active":"2021-01-29T13:52:24.188931+0000","last_became_peered":"2021-01-29T13:52:24.188931+0000","last_unstale":"2021-01-29T13:57:00.847683+0000","last_undegraded":"2021-01-29T13:57:00.847683+0000","last_fullsized":"2021-01-29T13:57:00.847683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:55:57.162808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:55:57.162808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"120'6449","reported_seq":"8595","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.652824+0000","last_change":"2021-01-29T13:55:45.625490+0000","last_active":"2021-01-29T13:57:00.652824+0000","last_peered":"2021-01-29T13:57:00.652824+0000","last_clean":"2021-01-29T13:57:00.652824+0000","last_became_active":"2021-01-29T13:52:39.830011+0000","last_became_peered":"2021-01-29T13:52:39.830011+0000","last_unstale":"2021-01-29T13:57:00.652824+0000","last_undegraded":"2021-01-29T13:57:00.652824+0000","last_fullsized":"2021-01-29T13:57:00.652824+0000","mapping_epoch":26,"log_start":"72'3403","ondisk_log_start":"72'3403","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'6250","last_scrub_stamp":"2021-01-29T13:55:45.625379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:38.811171+0000","last_clean_scrub_stamp":"2021-01-29T13:55:45.625379+0000","log_size":3046,"ondisk_log_size":3046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2928382292,"num_objects":986,"num_object_clones":0,"num_object_copies":1972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":2501,"num_read_kb":1252,"num_write":5879,"num_write_kb":1639641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":3315,"num_flush":761,"num_flush_kb":2192275,"num_evict":269,"num_evict_kb":760560,"num_promote":1249,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"112","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.552366+0000","last_change":"2021-01-29T13:55:42.661441+0000","last_active":"2021-01-29T13:57:00.552366+0000","last_peered":"2021-01-29T13:57:00.552366+0000","last_clean":"2021-01-29T13:57:00.552366+0000","last_became_active":"2021-01-29T13:53:29.493255+0000","last_became_peered":"2021-01-29T13:53:29.493255+0000","last_unstale":"2021-01-29T13:57:00.552366+0000","last_undegraded":"2021-01-29T13:57:00.552366+0000","last_fullsized":"2021-01-29T13:57:00.552366+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:55:42.661371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:55:42.661371+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3","version":"120'671","reported_seq":"1016","reported_epoch":"142","state":"active+remapped+backfilling","last_fresh":"2021-01-29T13:57:05.597246+0000","last_change":"2021-01-29T13:56:47.680232+0000","last_active":"2021-01-29T13:57:05.597246+0000","last_peered":"2021-01-29T13:57:05.597246+0000","last_clean":"2021-01-29T13:56:36.386610+0000","last_became_active":"2021-01-29T13:56:47.674484+0000","last_became_peered":"2021-01-29T13:56:47.674484+0000","last_unstale":"2021-01-29T13:57:05.597246+0000","last_undegraded":"2021-01-29T13:57:05.597246+0000","last_fullsized":"2021-01-29T13:57:05.597246+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'570","last_scrub_stamp":"2021-01-29T13:55:44.911108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:35.792587+0000","last_clean_scrub_stamp":"2021-01-29T13:55:44.911108+0000","log_size":671,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2038945493,"num_objects":671,"num_object_clones":0,"num_object_copies":2684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":590,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":671,"num_write_kb":1991482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":601,"num_bytes_recovered":1847436236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4,7],"acting":[6,2,4,7],"avail_no_missing":["6(0)","2(1)","4(2)","7(3)"],"object_location_counts":[{"shards":"2(1),4(2),6(0),7(3)","objects":671}],"blocked_by":[],"up_primary":0,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"142","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:01.550240+0000","last_change":"2021-01-29T13:57:01.550191+0000","last_active":"2021-01-29T13:57:01.550240+0000","last_peered":"2021-01-29T13:57:01.550240+0000","last_clean":"2021-01-29T13:57:01.550240+0000","last_became_active":"2021-01-29T13:52:23.765595+0000","last_became_peered":"2021-01-29T13:52:23.765595+0000","last_unstale":"2021-01-29T13:57:01.550240+0000","last_undegraded":"2021-01-29T13:57:01.550240+0000","last_fullsized":"2021-01-29T13:57:01.550240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:57:01.550087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:57:01.550087+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"117'710","reported_seq":"880","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.847676+0000","last_change":"2021-01-29T13:56:09.328006+0000","last_active":"2021-01-29T13:57:00.847676+0000","last_peered":"2021-01-29T13:57:00.847676+0000","last_clean":"2021-01-29T13:57:00.847676+0000","last_became_active":"2021-01-29T13:53:09.969063+0000","last_became_peered":"2021-01-29T13:53:09.969063+0000","last_unstale":"2021-01-29T13:57:00.847676+0000","last_undegraded":"2021-01-29T13:57:00.847676+0000","last_fullsized":"2021-01-29T13:57:00.847676+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'710","last_scrub_stamp":"2021-01-29T13:56:09.327903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:35.792587+0000","last_clean_scrub_stamp":"2021-01-29T13:56:09.327903+0000","log_size":710,"ondisk_log_size":710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2107977089,"num_objects":710,"num_object_clones":0,"num_object_copies":2840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":710,"num_write_kb":2058922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2,4],"acting":[5,7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"155","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.849258+0000","last_change":"2021-01-29T13:56:59.467725+0000","last_active":"2021-01-29T13:57:00.849258+0000","last_peered":"2021-01-29T13:57:00.849258+0000","last_clean":"2021-01-29T13:57:00.849258+0000","last_became_active":"2021-01-29T13:52:24.188578+0000","last_became_peered":"2021-01-29T13:52:24.188578+0000","last_unstale":"2021-01-29T13:57:00.849258+0000","last_undegraded":"2021-01-29T13:57:00.849258+0000","last_fullsized":"2021-01-29T13:57:00.849258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:56:59.467654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:56:59.467654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"142'4","reported_seq":"152","reported_epoch":"142","state":"active+clean","last_fresh":"2021-01-29T13:57:01.982145+0000","last_change":"2021-01-29T13:57:01.982145+0000","last_active":"2021-01-29T13:57:01.982145+0000","last_peered":"2021-01-29T13:57:01.982145+0000","last_clean":"2021-01-29T13:57:01.982145+0000","last_became_active":"2021-01-29T13:52:20.655015+0000","last_became_peered":"2021-01-29T13:52:20.655015+0000","last_unstale":"2021-01-29T13:57:01.982145+0000","last_undegraded":"2021-01-29T13:57:01.982145+0000","last_fullsized":"2021-01-29T13:57:01.982145+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'4","last_scrub_stamp":"2021-01-29T13:57:01.982073+0000","last_deep_scrub":"142'4","last_deep_scrub_stamp":"2021-01-29T13:57:01.982073+0000","last_clean_scrub_stamp":"2021-01-29T13:57:01.982073+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8249,"num_omap_keys":2,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"120'761","reported_seq":"2454","reported_epoch":"142","state":"active+remapped+backfilling","last_fresh":"2021-01-29T13:57:03.004043+0000","last_change":"2021-01-29T13:56:48.363777+0000","last_active":"2021-01-29T13:57:03.004043+0000","last_peered":"2021-01-29T13:57:03.004043+0000","last_clean":"2021-01-29T13:56:36.386422+0000","last_became_active":"2021-01-29T13:56:48.362095+0000","last_became_peered":"2021-01-29T13:56:48.362095+0000","last_unstale":"2021-01-29T13:57:03.004043+0000","last_undegraded":"2021-01-29T13:57:03.004043+0000","last_fullsized":"2021-01-29T13:57:03.004043+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'761","last_scrub_stamp":"2021-01-29T13:55:53.981049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:35.792587+0000","last_clean_scrub_stamp":"2021-01-29T13:55:53.981049+0000","log_size":761,"ondisk_log_size":761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2244508217,"num_objects":761,"num_object_clones":0,"num_object_copies":3044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":637,"num_objects_unfound":0,"num_objects_dirty":761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":761,"num_write_kb":2192275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":747,"num_bytes_recovered":2213000897,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7,0],"acting":[4,2,7,1],"avail_no_missing":["4(0)","1(3)","2(1)","7(2)"],"object_location_counts":[{"shards":"1(3),2(1),4(0),7(2)","objects":761}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"144","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.849237+0000","last_change":"2021-01-29T13:56:41.371116+0000","last_active":"2021-01-29T13:57:00.849237+0000","last_peered":"2021-01-29T13:57:00.849237+0000","last_clean":"2021-01-29T13:57:00.849237+0000","last_became_active":"2021-01-29T13:52:23.765671+0000","last_became_peered":"2021-01-29T13:52:23.765671+0000","last_unstale":"2021-01-29T13:57:00.849237+0000","last_undegraded":"2021-01-29T13:57:00.849237+0000","last_fullsized":"2021-01-29T13:57:00.849237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:56:41.371025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:56:41.371025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"118'720","reported_seq":"886","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.652827+0000","last_change":"2021-01-29T13:55:59.202455+0000","last_active":"2021-01-29T13:57:00.652827+0000","last_peered":"2021-01-29T13:57:00.652827+0000","last_clean":"2021-01-29T13:57:00.652827+0000","last_became_active":"2021-01-29T13:53:09.587252+0000","last_became_peered":"2021-01-29T13:53:09.587252+0000","last_unstale":"2021-01-29T13:57:00.652827+0000","last_undegraded":"2021-01-29T13:57:00.652827+0000","last_fullsized":"2021-01-29T13:57:00.652827+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'720","last_scrub_stamp":"2021-01-29T13:55:59.202349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:35.792587+0000","last_clean_scrub_stamp":"2021-01-29T13:55:59.202349+0000","log_size":720,"ondisk_log_size":720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2156154079,"num_objects":720,"num_object_clones":0,"num_object_copies":2880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":720,"num_write_kb":2105989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4,3],"acting":[6,2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"120'5986","reported_seq":"10252","reported_epoch":"142","state":"active+remapped+backfilling","last_fresh":"2021-01-29T13:57:05.668638+0000","last_change":"2021-01-29T13:56:47.680268+0000","last_active":"2021-01-29T13:57:05.668638+0000","last_peered":"2021-01-29T13:57:05.668638+0000","last_clean":"2021-01-29T13:56:36.386614+0000","last_became_active":"2021-01-29T13:56:47.673885+0000","last_became_peered":"2021-01-29T13:56:47.673885+0000","last_unstale":"2021-01-29T13:57:05.668638+0000","last_undegraded":"2021-01-29T13:57:05.668638+0000","last_fullsized":"2021-01-29T13:57:05.668638+0000","mapping_epoch":127,"log_start":"71'2904","ondisk_log_start":"71'2904","created":26,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'5955","last_scrub_stamp":"2021-01-29T13:55:51.275882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:38.811171+0000","last_clean_scrub_stamp":"2021-01-29T13:55:51.275882+0000","log_size":3082,"ondisk_log_size":3082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2947693900,"num_objects":985,"num_object_clones":0,"num_object_copies":1970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":849,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":2316,"num_read_kb":1158,"num_write":5435,"num_write_kb":1544566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1129,"num_bytes_recovered":1542062053,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":2771,"num_flush":671,"num_flush_kb":1991482,"num_evict":178,"num_evict_kb":526959,"num_promote":1172,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":985}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"157","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:01.451085+0000","last_change":"2021-01-29T13:57:01.451085+0000","last_active":"2021-01-29T13:57:01.451085+0000","last_peered":"2021-01-29T13:57:01.451085+0000","last_clean":"2021-01-29T13:57:01.451085+0000","last_became_active":"2021-01-29T13:53:09.377717+0000","last_became_peered":"2021-01-29T13:53:09.377717+0000","last_unstale":"2021-01-29T13:57:01.451085+0000","last_undegraded":"2021-01-29T13:57:01.451085+0000","last_fullsized":"2021-01-29T13:57:01.451085+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:55:33.131622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:55:33.131622+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"140","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.847720+0000","last_change":"2021-01-29T13:56:53.807344+0000","last_active":"2021-01-29T13:57:00.847720+0000","last_peered":"2021-01-29T13:57:00.847720+0000","last_clean":"2021-01-29T13:57:00.847720+0000","last_became_active":"2021-01-29T13:52:40.828334+0000","last_became_peered":"2021-01-29T13:52:40.828334+0000","last_unstale":"2021-01-29T13:57:00.847720+0000","last_undegraded":"2021-01-29T13:57:00.847720+0000","last_fullsized":"2021-01-29T13:57:00.847720+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:56:53.807251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:56:53.807251+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"117'6273","reported_seq":"8319","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.552349+0000","last_change":"2021-01-29T13:55:55.445357+0000","last_active":"2021-01-29T13:57:00.552349+0000","last_peered":"2021-01-29T13:57:00.552349+0000","last_clean":"2021-01-29T13:57:00.552349+0000","last_became_active":"2021-01-29T13:53:09.369472+0000","last_became_peered":"2021-01-29T13:53:09.369472+0000","last_unstale":"2021-01-29T13:57:00.552349+0000","last_undegraded":"2021-01-29T13:57:00.552349+0000","last_fullsized":"2021-01-29T13:57:00.552349+0000","mapping_epoch":42,"log_start":"71'3203","ondisk_log_start":"71'3203","created":26,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'6273","last_scrub_stamp":"2021-01-29T13:55:55.445229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:38.811171+0000","last_clean_scrub_stamp":"2021-01-29T13:55:55.445229+0000","log_size":3070,"ondisk_log_size":3070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2935339657,"num_objects":992,"num_object_clones":0,"num_object_copies":1984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":496,"num_whiteouts":0,"num_read":2414,"num_read_kb":1208,"num_write":5647,"num_write_kb":1593121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":3044,"num_flush":710,"num_flush_kb":2058922,"num_evict":219,"num_evict_kb":640347,"num_promote":1206,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"24","reported_epoch":"141","state":"active+clean","last_fresh":"2021-01-29T13:57:00.611252+0000","last_change":"2021-01-29T13:56:46.442031+0000","last_active":"2021-01-29T13:57:00.611252+0000","last_peered":"2021-01-29T13:57:00.611252+0000","last_clean":"2021-01-29T13:57:00.611252+0000","last_became_active":"2021-01-29T13:56:46.441589+0000","last_became_peered":"2021-01-29T13:56:46.441589+0000","last_unstale":"2021-01-29T13:57:00.611252+0000","last_undegraded":"2021-01-29T13:57:00.611252+0000","last_fullsized":"2021-01-29T13:57:00.611252+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:55:55.134762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:55:55.134762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"118'6270","reported_seq":"8351","reported_epoch":"142","state":"active+clean","last_fresh":"2021-01-29T13:57:04.235378+0000","last_change":"2021-01-29T13:57:04.235378+0000","last_active":"2021-01-29T13:57:04.235378+0000","last_peered":"2021-01-29T13:57:04.235378+0000","last_clean":"2021-01-29T13:57:04.235378+0000","last_became_active":"2021-01-29T13:52:39.823642+0000","last_became_peered":"2021-01-29T13:52:39.823642+0000","last_unstale":"2021-01-29T13:57:04.235378+0000","last_undegraded":"2021-01-29T13:57:04.235378+0000","last_fullsized":"2021-01-29T13:57:04.235378+0000","mapping_epoch":26,"log_start":"71'3205","ondisk_log_start":"71'3205","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6270","last_scrub_stamp":"2021-01-29T13:57:04.235268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:38.811171+0000","last_clean_scrub_stamp":"2021-01-29T13:57:04.235268+0000","log_size":3065,"ondisk_log_size":3065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2946120877,"num_objects":995,"num_object_clones":0,"num_object_copies":1990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":2420,"num_read_kb":1212,"num_write":5684,"num_write_kb":1601214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":3066,"num_flush":720,"num_flush_kb":2105989,"num_evict":218,"num_evict_kb":643724,"num_promote":1208,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":4,"stat_sum":{"num_bytes":11757536726,"num_objects":3958,"num_object_clones":0,"num_object_copies":7916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":849,"num_objects_unfound":0,"num_objects_dirty":1960,"num_whiteouts":0,"num_read":9651,"num_read_kb":4830,"num_write":22645,"num_write_kb":6378542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1129,"num_bytes_recovered":1542062053,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":21,"num_bytes_hit_set_archive":12196,"num_flush":2862,"num_flush_kb":8348668,"num_evict":884,"num_evict_kb":2571590,"num_promote":4835,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11918577664,"data_stored":11828415316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12263,"ondisk_log_size":12263,"up":8,"acting":8,"num_store_stats":6},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":8547584878,"num_objects":2862,"num_object_clones":0,"num_object_copies":11448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1227,"num_objects_unfound":0,"num_objects_dirty":2862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2862,"num_write_kb":8348668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1348,"num_bytes_recovered":4060437133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13299490816,"data_stored":18228555776,"data_compressed":505072361,"data_compressed_allocated":4929064960,"data_compressed_original":9858129920,"omap_allocated":0,"internal_metadata":0},"log_size":2862,"ondisk_log_size":2862,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8249,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":0,"up_from":124,"seq":532575944712,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1942776,"kb_used_data":1921328,"kb_used_omap":0,"kb_used_meta":21439,"kb_avail":92429064,"statfs":{"total":96636764160,"available":94647361536,"internally_reserved":0,"allocated":1967439872,"data_stored":2257930354,"data_compressed":30754208,"data_compressed_allocated":299577344,"data_compressed_original":599154688,"omap_allocated":972,"internal_metadata":21953588},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.23}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98799999999999999}]}]},{"osd":7,"up_from":15,"seq":64424509501,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2311348,"kb_used_data":2281004,"kb_used_omap":0,"kb_used_meta":30336,"kb_avail":92060492,"statfs":{"total":96636764160,"available":94269943808,"internally_reserved":0,"allocated":2335748096,"data_stored":3200465622,"data_compressed":88587048,"data_compressed_allocated":865042432,"data_compressed_original":1730084864,"omap_allocated":0,"internal_metadata":31064064},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:54:26 2021","interfaces":[{"interface":"back","average":{"1min":51.597999999999999,"5min":29.779,"15min":26.143000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":906.00800000000004,"5min":906.00800000000004,"15min":906.00800000000004},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.1040000000000001,"5min":0.95499999999999996,"15min":0.93000000000000005},"min":{"1min":0.61499999999999999,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":4.0259999999999998,"5min":4.0259999999999998,"15min":4.0259999999999998},"last":1.3089999999999999}]},{"osd":1,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.3340000000000001,"15min":1.49},"min":{"1min":0.65800000000000003,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.4410000000000001,"5min":11.195,"15min":11.195},"last":1.272},{"interface":"front","average":{"1min":1.006,"5min":23.977,"15min":32.523000000000003},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.3759999999999999,"5min":777.02499999999998,"15min":777.02499999999998},"last":1.581}]},{"osd":2,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":46.633000000000003,"5min":10.444000000000001,"15min":4.5460000000000003},"min":{"1min":0.62,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":512.423,"5min":512.423,"15min":512.423},"last":0.999},{"interface":"front","average":{"1min":1.1559999999999999,"5min":2.1200000000000001,"15min":1.774},"min":{"1min":0.79300000000000004,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.472,"5min":60.529000000000003,"15min":60.529000000000003},"last":1.3759999999999999}]},{"osd":3,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":10.481,"5min":3.0230000000000001,"15min":1.746},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":211.47900000000001,"5min":211.47900000000001,"15min":211.47900000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.46,"15min":1.607},"min":{"1min":0.92300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.4930000000000001,"5min":11.146000000000001,"15min":11.146000000000001},"last":1.4079999999999999}]},{"osd":4,"last update":"Fri Jan 29 13:56:16 2021","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.264,"15min":1.3380000000000001},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.4299999999999999,"5min":5.8449999999999998,"15min":5.8449999999999998},"last":1.343},{"interface":"front","average":{"1min":1.0229999999999999,"5min":14.401999999999999,"15min":20.280000000000001},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.4399999999999999,"5min":391.66699999999997,"15min":391.66699999999997},"last":1.54}]},{"osd":5,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":3.4470000000000001,"15min":2.2829999999999999},"min":{"1min":0.74199999999999999,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.601,"5min":175.83199999999999,"15min":175.83199999999999},"last":1.625},{"interface":"front","average":{"1min":1.105,"5min":1.2549999999999999,"15min":1.2},"min":{"1min":0.77600000000000002,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.526,"5min":4.4020000000000001,"15min":4.4020000000000001},"last":1.6020000000000001}]},{"osd":6,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.5649999999999999,"15min":1.536},"min":{"1min":0.84199999999999997,"5min":0.754,"15min":0.754},"max":{"1min":1.5269999999999999,"5min":10.063000000000001,"15min":10.063000000000001},"last":1.647},{"interface":"front","average":{"1min":1.123,"5min":14.808,"15min":27.946000000000002},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.4690000000000001,"5min":623.01999999999998,"15min":623.01999999999998},"last":1.754}]}]},{"osd":2,"up_from":15,"seq":64424509500,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5736668,"kb_used_data":5682580,"kb_used_omap":0,"kb_used_meta":54080,"kb_avail":88635172,"statfs":{"total":96636764160,"available":90762416128,"internally_reserved":0,"allocated":5818961920,"data_stored":6956451332,"data_compressed":118712623,"data_compressed_allocated":1158184960,"data_compressed_original":2316369920,"omap_allocated":0,"internal_metadata":55377920},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:54:22 2021","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":29.140000000000001,"15min":33.799999999999997},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":2.012,"5min":565.52099999999996,"15min":565.52099999999996},"last":0.997},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.0209999999999999,"15min":0.94299999999999995},"min":{"1min":0.72499999999999998,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":3.6030000000000002,"5min":3.6030000000000002,"15min":3.6030000000000002},"last":0.77000000000000002}]},{"osd":1,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":40.582999999999998,"5min":17.475000000000001,"15min":6.4610000000000003},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":676.39099999999996,"5min":721.11000000000001,"15min":721.11000000000001},"last":1.0489999999999999},{"interface":"front","average":{"1min":0.89700000000000002,"5min":21.452999999999999,"15min":7.8179999999999996},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.1479999999999999,"5min":1000.376,"15min":1000.376},"last":0.81299999999999994}]},{"osd":3,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":51.539000000000001,"5min":11.381,"15min":4.3440000000000003},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":501.61200000000002,"5min":501.61200000000002,"15min":501.61200000000002},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.85399999999999998,"5min":1.1890000000000001,"15min":1.0640000000000001},"min":{"1min":0.57399999999999995,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.6839999999999999,"5min":6.1870000000000003,"15min":6.1870000000000003},"last":0.78900000000000003}]},{"osd":4,"last update":"Fri Jan 29 13:56:15 2021","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.478,"15min":1.53},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.6719999999999999,"5min":3.694,"15min":3.694},"last":1.6160000000000001},{"interface":"front","average":{"1min":1.155,"5min":1.407,"15min":1.3859999999999999},"min":{"1min":0.871,"5min":0.871,"15min":0.871},"max":{"1min":1.7509999999999999,"5min":3.7189999999999999,"15min":3.7189999999999999},"last":1.6479999999999999}]},{"osd":5,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":21.327999999999999,"5min":5.8109999999999999,"15min":2.7749999999999999},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":183.761,"5min":183.761,"15min":183.761},"last":0.86199999999999999},{"interface":"front","average":{"1min":41.545999999999999,"5min":9.516,"15min":4.0499999999999998},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":521.02499999999998,"5min":521.02499999999998,"15min":521.02499999999998},"last":1.0289999999999999}]},{"osd":6,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.3720000000000001,"15min":1.3009999999999999},"min":{"1min":0.89900000000000002,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.474,"5min":3.46,"15min":3.46},"last":1.544},{"interface":"front","average":{"1min":1.111,"5min":1.4450000000000001,"15min":1.2809999999999999},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.4570000000000001,"5min":7.0209999999999999,"15min":7.0209999999999999},"last":0.88200000000000001}]},{"osd":7,"last update":"Fri Jan 29 13:56:28 2021","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":8.5039999999999996,"15min":3.754},"min":{"1min":0.99099999999999999,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":1.7050000000000001,"5min":591.18899999999996,"15min":591.18899999999996},"last":1.5820000000000001},{"interface":"front","average":{"1min":52.493000000000002,"5min":11.779999999999999,"15min":4.8220000000000001},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":923.40099999999995,"5min":923.40099999999995,"15min":923.40099999999995},"last":0.95499999999999996}]}]},{"osd":1,"up_from":15,"seq":64424509501,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2154844,"kb_used_data":2115092,"kb_used_omap":0,"kb_used_meta":39744,"kb_avail":92216996,"statfs":{"total":96636764160,"available":94430203904,"internally_reserved":0,"allocated":2165854208,"data_stored":2457969146,"data_compressed":31021897,"data_compressed_allocated":302632960,"data_compressed_original":605265920,"omap_allocated":0,"internal_metadata":40697856},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:54:25 2021","interfaces":[{"interface":"back","average":{"1min":1.7629999999999999,"5min":1.0349999999999999,"15min":0.91300000000000003},"min":{"1min":0.63300000000000001,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":5.0629999999999997,"5min":5.0629999999999997,"15min":5.0629999999999997},"last":0.92400000000000004},{"interface":"front","average":{"1min":39.090000000000003,"5min":8.5060000000000002,"15min":3.4079999999999999},"min":{"1min":0.58699999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":752.73400000000004,"5min":752.73400000000004,"15min":752.73400000000004},"last":0.54300000000000004}]},{"osd":2,"last update":"Fri Jan 29 13:56:29 2021","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":22.648,"15min":29.776},"min":{"1min":0.55100000000000005,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":4.2629999999999999,"5min":832.21600000000001,"15min":832.21600000000001},"last":0.83499999999999996},{"interface":"front","average":{"1min":5.2699999999999996,"5min":11.266,"15min":4.4199999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":84.277000000000001,"5min":899.43399999999997,"15min":899.43399999999997},"last":0.86599999999999999}]},{"osd":3,"last update":"Fri Jan 29 13:56:29 2021","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":1.306,"15min":1.099},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":2.1499999999999999,"5min":6.0579999999999998,"15min":6.0579999999999998},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":5.548,"15min":9.8919999999999995},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.5489999999999999,"5min":244.60900000000001,"15min":244.60900000000001},"last":0.68899999999999995}]},{"osd":4,"last update":"Fri Jan 29 13:56:15 2021","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.4610000000000001,"15min":1.633},"min":{"1min":0.63800000000000001,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.5960000000000001,"5min":5.242,"15min":5.242},"last":0.71399999999999997},{"interface":"front","average":{"1min":1.202,"5min":1.9259999999999999,"15min":2.1680000000000001},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.5760000000000001,"5min":6.2590000000000003,"15min":6.2590000000000003},"last":1.024}]},{"osd":5,"last update":"Fri Jan 29 13:56:29 2021","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":14.667999999999999,"15min":5.694},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":2.0350000000000001,"5min":701.73400000000004,"15min":701.73400000000004},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.133,"5min":1.3400000000000001,"15min":1.26},"min":{"1min":0.75700000000000001,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.917,"5min":5.758,"15min":5.758},"last":0.76600000000000001}]},{"osd":6,"last update":"Fri Jan 29 13:56:29 2021","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.4419999999999999,"15min":1.3440000000000001},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.9139999999999999,"5min":5.7199999999999998,"15min":5.7199999999999998},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.258,"5min":4.0110000000000001,"15min":6.2149999999999999},"min":{"1min":0.72299999999999998,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.6230000000000002,"5min":132.35300000000001,"15min":132.35300000000001},"last":1.0589999999999999}]},{"osd":7,"last update":"Fri Jan 29 13:56:29 2021","interfaces":[{"interface":"back","average":{"1min":37.829999999999998,"5min":8.8330000000000002,"15min":3.823},"min":{"1min":0.81299999999999994,"5min":0.752,"15min":0.752},"max":{"1min":696.96400000000006,"5min":696.96400000000006,"15min":696.96400000000006},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.4990000000000001,"15min":1.413},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.0369999999999999,"5min":5.9450000000000003,"15min":5.9450000000000003},"last":0.85199999999999998}]}]},{"osd":3,"up_from":15,"seq":64424509501,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3443244,"kb_used_data":3391652,"kb_used_omap":0,"kb_used_meta":51584,"kb_avail":90928596,"statfs":{"total":96636764160,"available":93110882304,"internally_reserved":0,"allocated":3473051648,"data_stored":3742620723,"data_compressed":29735076,"data_compressed_allocated":290086912,"data_compressed_original":580173824,"omap_allocated":0,"internal_metadata":52822016},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:54:26 2021","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":0.93999999999999995,"15min":0.90400000000000003},"min":{"1min":0.67500000000000004,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.617,"5min":2.617,"15min":2.617},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.135,"5min":3.3719999999999999,"15min":3.7450000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":3.9620000000000002,"5min":60.329000000000001,"15min":60.329000000000001},"last":1.0529999999999999}]},{"osd":1,"last update":"Fri Jan 29 13:56:30 2021","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.97599999999999998,"15min":0.91300000000000003},"min":{"1min":0.497,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0109999999999999,"5min":2.5819999999999999,"15min":2.5819999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.82699999999999996,"5min":10.324,"15min":4.0970000000000004},"min":{"1min":0.63800000000000001,"5min":0.496,"15min":0.496},"max":{"1min":1.0629999999999999,"5min":977.78200000000004,"15min":977.78200000000004},"last":0.66900000000000004}]},{"osd":2,"last update":"Fri Jan 29 13:56:30 2021","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":1.208,"15min":1.0880000000000001},"min":{"1min":0.61099999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.22,"5min":5.7539999999999996,"15min":5.7539999999999996},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.018,"5min":1.383,"15min":1.4339999999999999},"min":{"1min":0.73199999999999998,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.4870000000000001,"5min":6.4509999999999996,"15min":6.4509999999999996},"last":0.96899999999999997}]},{"osd":4,"last update":"Fri Jan 29 13:56:11 2021","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":11.106,"15min":4.5620000000000003},"min":{"1min":0.70799999999999996,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.3979999999999999,"5min":916.09500000000003,"15min":916.09500000000003},"last":0.71099999999999997},{"interface":"front","average":{"1min":1.1479999999999999,"5min":11.696999999999999,"15min":4.8819999999999997},"min":{"1min":0.82099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.593,"5min":977.91499999999996,"15min":977.91499999999996},"last":0.92900000000000005}]},{"osd":5,"last update":"Fri Jan 29 13:56:30 2021","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":6.9089999999999998,"15min":3.1429999999999998},"min":{"1min":0.72799999999999998,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.8160000000000001,"5min":503.93000000000001,"15min":503.93000000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.077,"5min":6.79,"15min":3.0230000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.3799999999999999,"5min":503.74299999999999,"15min":503.74299999999999},"last":1.141}]},{"osd":6,"last update":"Fri Jan 29 13:56:30 2021","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.4490000000000001,"15min":1.351},"min":{"1min":0.874,"5min":0.751,"15min":0.751},"max":{"1min":3.2930000000000001,"5min":4.7539999999999996,"15min":4.7539999999999996},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.339,"5min":19.396999999999998,"15min":37.238999999999997},"min":{"1min":0.876,"5min":0.876,"15min":0.876},"max":{"1min":3.3690000000000002,"5min":851.89099999999996,"15min":851.89099999999996},"last":1.173}]},{"osd":7,"last update":"Fri Jan 29 13:56:30 2021","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.5609999999999999,"15min":1.5069999999999999},"min":{"1min":0.90800000000000003,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.613,"5min":6.4240000000000004,"15min":6.4240000000000004},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.8300000000000001,"15min":1.966},"min":{"1min":1.0629999999999999,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.635,"5min":15.597,"15min":15.597},"last":1.038}]}]},{"osd":4,"up_from":42,"seq":180388626482,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4411588,"kb_used_data":4363992,"kb_used_omap":0,"kb_used_meta":47423,"kb_avail":89960252,"statfs":{"total":96636764160,"available":92119298048,"internally_reserved":0,"allocated":4468727808,"data_stored":5613881037,"data_compressed":118370768,"data_compressed_allocated":1155686400,"data_compressed_original":2311372800,"omap_allocated":972,"internal_metadata":48561204},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:54:12 2021","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":4.2450000000000001,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.746},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":4.1589999999999998,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":1.141}]},{"osd":1,"last update":"Fri Jan 29 13:56:18 2021","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":43.862000000000002,"15min":62.893000000000001},"min":{"1min":0.76800000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":2.6850000000000001,"5min":924.63300000000004,"15min":924.63300000000004},"last":1.008},{"interface":"front","average":{"1min":74.046000000000006,"5min":15.648,"15min":5.9059999999999997},"min":{"1min":0.84299999999999997,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":898.86900000000003,"5min":898.86900000000003,"15min":898.86900000000003},"last":1.2090000000000001}]},{"osd":2,"last update":"Fri Jan 29 13:56:18 2021","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":11.952,"15min":4.7160000000000002},"min":{"1min":0.745,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.3149999999999999,"5min":857.78899999999999,"15min":857.78899999999999},"last":1.1679999999999999},{"interface":"front","average":{"1min":44.206000000000003,"5min":9.843,"15min":4.0510000000000002},"min":{"1min":0.68600000000000005,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":779.38400000000001,"5min":779.38400000000001,"15min":779.38400000000001},"last":0.78100000000000003}]},{"osd":3,"last update":"Fri Jan 29 13:56:18 2021","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":12.48,"15min":14.343},"min":{"1min":0.754,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":2.738,"5min":311.68299999999999,"15min":311.68299999999999},"last":1.079},{"interface":"front","average":{"1min":1.028,"5min":1.236,"15min":1.264},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.4359999999999999,"5min":3.6539999999999999,"15min":3.6539999999999999},"last":1.2509999999999999}]},{"osd":5,"last update":"Fri Jan 29 13:56:18 2021","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":3.7309999999999999,"15min":1.964},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.5109999999999999,"5min":239.99199999999999,"15min":239.99199999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.171,"15min":1.1599999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":3.1749999999999998,"5min":3.1749999999999998,"15min":3.1749999999999998},"last":1.03}]},{"osd":6,"last update":"Fri Jan 29 13:56:18 2021","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.238,"15min":1.206},"min":{"1min":0.72499999999999998,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":2.7789999999999999,"5min":5.5890000000000004,"15min":5.5890000000000004},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.252,"15min":1.296},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.657,"5min":3.6880000000000002,"15min":3.6880000000000002},"last":0.91600000000000004}]},{"osd":7,"last update":"Fri Jan 29 13:56:18 2021","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.3,"15min":1.286},"min":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":2.988,"5min":3.0169999999999999,"15min":3.0169999999999999},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.3380000000000001,"15min":1.3400000000000001},"min":{"1min":0.96099999999999997,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":1.635,"5min":3.6699999999999999,"15min":3.6699999999999999},"last":1.357}]}]},{"osd":5,"up_from":15,"seq":64424509500,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":765812,"kb_used_data":751852,"kb_used_omap":0,"kb_used_meta":13952,"kb_avail":93606028,"statfs":{"total":96636764160,"available":95852572672,"internally_reserved":0,"allocated":769896448,"data_stored":1055801421,"data_compressed":29380114,"data_compressed_allocated":286253056,"data_compressed_original":572506112,"omap_allocated":0,"internal_metadata":14286848},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:54:20 2021","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.83999999999999997,"15min":0.81299999999999994},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":2.3250000000000002,"5min":2.3250000000000002,"15min":2.3250000000000002},"last":1.079},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.101,"15min":1.044},"min":{"1min":0.65600000000000003,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":4.8410000000000002,"5min":4.8410000000000002,"15min":4.8410000000000002},"last":1.306}]},{"osd":1,"last update":"Fri Jan 29 13:56:20 2021","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":0.93600000000000005,"15min":0.85999999999999999},"min":{"1min":0.68500000000000005,"5min":0.498,"15min":0.498},"max":{"1min":2.3690000000000002,"5min":3.1819999999999999,"15min":3.1819999999999999},"last":1.1140000000000001},{"interface":"front","average":{"1min":51.244999999999997,"5min":11.042999999999999,"15min":4.2709999999999999},"min":{"1min":0.82999999999999996,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":732.80700000000002,"5min":732.80700000000002,"15min":732.80700000000002},"last":1.365}]},{"osd":2,"last update":"Fri Jan 29 13:56:20 2021","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":8.1699999999999999,"15min":3.548},"min":{"1min":0.73599999999999999,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.161,"5min":649.82899999999995,"15min":649.82899999999995},"last":1.6950000000000001},{"interface":"front","average":{"1min":1.095,"5min":1.1180000000000001,"15min":1.054},"min":{"1min":0.70699999999999996,"5min":0.623,"15min":0.623},"max":{"1min":2.4169999999999998,"5min":4.1799999999999997,"15min":4.1799999999999997},"last":1.145}]},{"osd":3,"last update":"Fri Jan 29 13:56:20 2021","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":1.137,"15min":1.133},"min":{"1min":0.626,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.2829999999999999,"5min":4.0890000000000004,"15min":4.0890000000000004},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.274,"15min":1.23},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":2.1200000000000001,"5min":4.827,"15min":4.827},"last":1.7350000000000001}]},{"osd":4,"last update":"Fri Jan 29 13:56:17 2021","interfaces":[{"interface":"back","average":{"1min":28.036000000000001,"5min":16.495999999999999,"15min":17.274999999999999},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":514.74199999999996,"5min":514.74199999999996,"15min":514.74199999999996},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.2430000000000001,"15min":1.204},"min":{"1min":0.86699999999999999,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":2.266,"5min":4.4359999999999999,"15min":4.4359999999999999},"last":1.7689999999999999}]},{"osd":6,"last update":"Fri Jan 29 13:56:20 2021","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":9.5289999999999999,"15min":4.0229999999999997},"min":{"1min":0.81499999999999995,"5min":0.622,"15min":0.622},"max":{"1min":2.3039999999999998,"5min":520.71799999999996,"15min":520.71799999999996},"last":1.804},{"interface":"front","average":{"1min":1.345,"5min":1.421,"15min":1.256},"min":{"1min":0.84399999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.3929999999999998,"5min":4.8700000000000001,"15min":4.8700000000000001},"last":1.8320000000000001}]},{"osd":7,"last update":"Fri Jan 29 13:56:20 2021","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.4710000000000001,"15min":1.5209999999999999},"min":{"1min":0.92700000000000005,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.3420000000000001,"5min":9.0109999999999992,"15min":9.0109999999999992},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.383,"5min":1.7270000000000001,"15min":1.7849999999999999},"min":{"1min":1.0049999999999999,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":2.3319999999999999,"5min":9.593,"15min":9.593},"last":1.96}]}]},{"osd":6,"up_from":15,"seq":64424509500,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4180688,"kb_used_data":4117640,"kb_used_omap":0,"kb_used_meta":63040,"kb_avail":90191152,"statfs":{"total":96636764160,"available":92355739648,"internally_reserved":0,"allocated":4216463360,"data_stored":4766148518,"data_compressed":58393750,"data_compressed_allocated":570441728,"data_compressed_original":1140883456,"omap_allocated":0,"internal_metadata":64552960},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:54:20 2021","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.379,"15min":1.381},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":3.0899999999999999,"5min":5.8380000000000001,"15min":5.8380000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":1.286,"5min":1.012,"15min":0.96599999999999997},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":7.3810000000000002,"5min":7.3810000000000002,"15min":7.3810000000000002},"last":0.50800000000000001}]},{"osd":1,"last update":"Fri Jan 29 13:56:22 2021","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":15.146000000000001,"15min":29.238},"min":{"1min":0.65400000000000003,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.353,"5min":684.42700000000002,"15min":684.42700000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.2849999999999999,"15min":1.1699999999999999},"min":{"1min":0.58899999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.306,"5min":6.1360000000000001,"15min":6.1360000000000001},"last":0.755}]},{"osd":2,"last update":"Fri Jan 29 13:56:22 2021","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.1919999999999999,"15min":1.242},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.492,"5min":7.2329999999999997,"15min":7.2329999999999997},"last":1.133},{"interface":"front","average":{"1min":36.773000000000003,"5min":8.3480000000000008,"15min":3.6339999999999999},"min":{"1min":0.68799999999999994,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":607.71000000000004,"5min":607.71000000000004,"15min":607.71000000000004},"last":0.67700000000000005}]},{"osd":3,"last update":"Fri Jan 29 13:56:22 2021","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":1.345,"15min":1.2470000000000001},"min":{"1min":0.66800000000000004,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.1779999999999999,"5min":6.048,"15min":6.048},"last":200.50399999999999},{"interface":"front","average":{"1min":29.335999999999999,"5min":6.8609999999999998,"15min":3.0449999999999999},"min":{"1min":0.83699999999999997,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":480.87299999999999,"5min":480.87299999999999,"15min":480.87299999999999},"last":1.21}]},{"osd":4,"last update":"Fri Jan 29 13:56:12 2021","interfaces":[{"interface":"back","average":{"1min":17.359999999999999,"5min":4.6619999999999999,"15min":2.456},"min":{"1min":0.629,"5min":0.62,"15min":0.62},"max":{"1min":263.584,"5min":263.584,"15min":263.584},"last":1.173},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.3069999999999999,"15min":1.417},"min":{"1min":0.78900000000000003,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.474,"5min":7.3449999999999998,"15min":7.3449999999999998},"last":0.82699999999999996}]},{"osd":5,"last update":"Fri Jan 29 13:56:22 2021","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.2969999999999999,"15min":1.349},"min":{"1min":0.86499999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.496,"5min":4.0069999999999997,"15min":4.0069999999999997},"last":1.034},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.6100000000000001,"15min":1.506},"min":{"1min":0.96499999999999997,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":1.3759999999999999,"5min":7.3140000000000001,"15min":7.3140000000000001},"last":1.052}]},{"osd":7,"last update":"Fri Jan 29 13:56:22 2021","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.4159999999999999,"15min":1.3380000000000001},"min":{"1min":0.90100000000000002,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.655,"5min":10.795,"15min":10.795},"last":1.375},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.3879999999999999,"15min":1.3919999999999999},"min":{"1min":0.85899999999999999,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.609,"5min":4.0579999999999998,"15min":4.0579999999999998},"last":0.99299999999999999}]}]}]}} , 2021-01-29T13:57:11.433 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-29T13:57:11.716 INFO:teuthology.orchestra.run.smithi183.stdout: 2021-01-30T06:14:58.017 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T13:57:44.175 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":301,"stamp":"2021-01-29T13:57:42.675417+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":20305121604,"num_objects":6822,"num_object_clones":0,"num_object_copies":19368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":632,"num_objects_unfound":0,"num_objects_dirty":4824,"num_whiteouts":0,"num_read":9651,"num_read_kb":4830,"num_write":25509,"num_write_kb":14727220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3207,"num_bytes_recovered":7385874976,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":21,"num_bytes_hit_set_archive":12196,"num_flush":2862,"num_flush_kb":8348668,"num_evict":884,"num_evict_kb":2571590,"num_promote":4835,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8249,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15129,"ondisk_log_size":15129,"up":42,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":45,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":25551644,"kb_used_data":25207644,"kb_used_omap":1,"kb_used_meta":343934,"kb_avail":729423076,"statfs":{"total":773094113280,"available":746929229824,"internally_reserved":0,"allocated":25812627456,"data_stored":30826570098,"data_compressed":523387886,"data_compressed_allocated":5107855360,"data_compressed_original":10215710720,"omap_allocated":1944,"internal_metadata":352188520},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-252,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":339084355,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.225468"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":"156","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.458800+0000","last_change":"2021-01-29T13:57:34.166076+0000","last_active":"2021-01-29T13:57:37.458800+0000","last_peered":"2021-01-29T13:57:37.458800+0000","last_clean":"2021-01-29T13:57:37.458800+0000","last_became_active":"2021-01-29T13:52:24.188931+0000","last_became_peered":"2021-01-29T13:52:24.188931+0000","last_unstale":"2021-01-29T13:57:37.458800+0000","last_undegraded":"2021-01-29T13:57:37.458800+0000","last_fullsized":"2021-01-29T13:57:37.458800+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:55:57.162808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:55:57.162808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"120'6449","reported_seq":"8600","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.461338+0000","last_change":"2021-01-29T13:55:45.625490+0000","last_active":"2021-01-29T13:57:37.461338+0000","last_peered":"2021-01-29T13:57:37.461338+0000","last_clean":"2021-01-29T13:57:37.461338+0000","last_became_active":"2021-01-29T13:52:39.830011+0000","last_became_peered":"2021-01-29T13:52:39.830011+0000","last_unstale":"2021-01-29T13:57:37.461338+0000","last_undegraded":"2021-01-29T13:57:37.461338+0000","last_fullsized":"2021-01-29T13:57:37.461338+0000","mapping_epoch":26,"log_start":"72'3403","ondisk_log_start":"72'3403","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'6250","last_scrub_stamp":"2021-01-29T13:55:45.625379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:38.811171+0000","last_clean_scrub_stamp":"2021-01-29T13:55:45.625379+0000","log_size":3046,"ondisk_log_size":3046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2928382292,"num_objects":986,"num_object_clones":0,"num_object_copies":1972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":2501,"num_read_kb":1252,"num_write":5879,"num_write_kb":1639641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":3315,"num_flush":761,"num_flush_kb":2192275,"num_evict":269,"num_evict_kb":760560,"num_promote":1249,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"117","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.497927+0000","last_change":"2021-01-29T13:55:42.661441+0000","last_active":"2021-01-29T13:57:37.497927+0000","last_peered":"2021-01-29T13:57:37.497927+0000","last_clean":"2021-01-29T13:57:37.497927+0000","last_became_active":"2021-01-29T13:53:29.493255+0000","last_became_peered":"2021-01-29T13:53:29.493255+0000","last_unstale":"2021-01-29T13:57:37.497927+0000","last_undegraded":"2021-01-29T13:57:37.497927+0000","last_fullsized":"2021-01-29T13:57:37.497927+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:55:42.661371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:55:42.661371+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3","version":"120'671","reported_seq":"1393","reported_epoch":"147","state":"active+remapped+backfilling","last_fresh":"2021-01-29T13:57:40.662249+0000","last_change":"2021-01-29T13:56:47.680232+0000","last_active":"2021-01-29T13:57:40.662249+0000","last_peered":"2021-01-29T13:57:40.662249+0000","last_clean":"2021-01-29T13:56:36.386610+0000","last_became_active":"2021-01-29T13:56:47.674484+0000","last_became_peered":"2021-01-29T13:56:47.674484+0000","last_unstale":"2021-01-29T13:57:40.662249+0000","last_undegraded":"2021-01-29T13:57:40.662249+0000","last_fullsized":"2021-01-29T13:57:40.662249+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'570","last_scrub_stamp":"2021-01-29T13:55:44.911108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:35.792587+0000","last_clean_scrub_stamp":"2021-01-29T13:55:44.911108+0000","log_size":671,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2038945493,"num_objects":671,"num_object_clones":0,"num_object_copies":2684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":404,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":671,"num_write_kb":1991482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":787,"num_bytes_recovered":2413500764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4,7],"acting":[6,2,4,7],"avail_no_missing":["6(0)","2(1)","4(2)","7(3)"],"object_location_counts":[{"shards":"2(1),4(2),6(0),7(3)","objects":671}],"blocked_by":[],"up_primary":0,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"147","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.457918+0000","last_change":"2021-01-29T13:57:01.550191+0000","last_active":"2021-01-29T13:57:37.457918+0000","last_peered":"2021-01-29T13:57:37.457918+0000","last_clean":"2021-01-29T13:57:37.457918+0000","last_became_active":"2021-01-29T13:52:23.765595+0000","last_became_peered":"2021-01-29T13:52:23.765595+0000","last_unstale":"2021-01-29T13:57:37.457918+0000","last_undegraded":"2021-01-29T13:57:37.457918+0000","last_fullsized":"2021-01-29T13:57:37.457918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:57:01.550087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:57:01.550087+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"117'710","reported_seq":"887","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.458693+0000","last_change":"2021-01-29T13:57:37.153012+0000","last_active":"2021-01-29T13:57:37.458693+0000","last_peered":"2021-01-29T13:57:37.458693+0000","last_clean":"2021-01-29T13:57:37.458693+0000","last_became_active":"2021-01-29T13:53:09.969063+0000","last_became_peered":"2021-01-29T13:53:09.969063+0000","last_unstale":"2021-01-29T13:57:37.458693+0000","last_undegraded":"2021-01-29T13:57:37.458693+0000","last_fullsized":"2021-01-29T13:57:37.458693+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'710","last_scrub_stamp":"2021-01-29T13:56:09.327903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:35.792587+0000","last_clean_scrub_stamp":"2021-01-29T13:56:09.327903+0000","log_size":710,"ondisk_log_size":710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2107977089,"num_objects":710,"num_object_clones":0,"num_object_copies":2840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":710,"num_write_kb":2058922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2,4],"acting":[5,7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"159","reported_epoch":"145","state":"active+clean","last_fresh":"2021-01-29T13:57:36.794315+0000","last_change":"2021-01-29T13:56:59.467725+0000","last_active":"2021-01-29T13:57:36.794315+0000","last_peered":"2021-01-29T13:57:36.794315+0000","last_clean":"2021-01-29T13:57:36.794315+0000","last_became_active":"2021-01-29T13:52:24.188578+0000","last_became_peered":"2021-01-29T13:52:24.188578+0000","last_unstale":"2021-01-29T13:57:36.794315+0000","last_undegraded":"2021-01-29T13:57:36.794315+0000","last_fullsized":"2021-01-29T13:57:36.794315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:56:59.467654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:56:59.467654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"142'4","reported_seq":"157","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.458197+0000","last_change":"2021-01-29T13:57:01.982145+0000","last_active":"2021-01-29T13:57:37.458197+0000","last_peered":"2021-01-29T13:57:37.458197+0000","last_clean":"2021-01-29T13:57:37.458197+0000","last_became_active":"2021-01-29T13:52:20.655015+0000","last_became_peered":"2021-01-29T13:52:20.655015+0000","last_unstale":"2021-01-29T13:57:37.458197+0000","last_undegraded":"2021-01-29T13:57:37.458197+0000","last_fullsized":"2021-01-29T13:57:37.458197+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'4","last_scrub_stamp":"2021-01-29T13:57:01.982073+0000","last_deep_scrub":"142'4","last_deep_scrub_stamp":"2021-01-29T13:57:01.982073+0000","last_clean_scrub_stamp":"2021-01-29T13:57:01.982073+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8249,"num_omap_keys":2,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"120'761","reported_seq":"3071","reported_epoch":"147","state":"active+remapped+backfilling","last_fresh":"2021-01-29T13:57:41.858978+0000","last_change":"2021-01-29T13:56:48.363777+0000","last_active":"2021-01-29T13:57:41.858978+0000","last_peered":"2021-01-29T13:57:41.858978+0000","last_clean":"2021-01-29T13:56:36.386422+0000","last_became_active":"2021-01-29T13:56:48.362095+0000","last_became_peered":"2021-01-29T13:56:48.362095+0000","last_unstale":"2021-01-29T13:57:41.858978+0000","last_undegraded":"2021-01-29T13:57:41.858978+0000","last_fullsized":"2021-01-29T13:57:41.858978+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'761","last_scrub_stamp":"2021-01-29T13:55:53.981049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:35.792587+0000","last_clean_scrub_stamp":"2021-01-29T13:55:53.981049+0000","log_size":761,"ondisk_log_size":761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2244508217,"num_objects":761,"num_object_clones":0,"num_object_copies":3044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":228,"num_objects_unfound":0,"num_objects_dirty":761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":761,"num_write_kb":2192275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1053,"num_bytes_recovered":3106526858,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7,0],"acting":[4,2,7,1],"avail_no_missing":["4(0)","1(3)","2(1)","7(2)"],"object_location_counts":[{"shards":"1(3),2(1),4(0),7(2)","objects":761}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"148","reported_epoch":"145","state":"active+clean","last_fresh":"2021-01-29T13:57:36.794261+0000","last_change":"2021-01-29T13:56:41.371116+0000","last_active":"2021-01-29T13:57:36.794261+0000","last_peered":"2021-01-29T13:57:36.794261+0000","last_clean":"2021-01-29T13:57:36.794261+0000","last_became_active":"2021-01-29T13:52:23.765671+0000","last_became_peered":"2021-01-29T13:52:23.765671+0000","last_unstale":"2021-01-29T13:57:36.794261+0000","last_undegraded":"2021-01-29T13:57:36.794261+0000","last_fullsized":"2021-01-29T13:57:36.794261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:56:41.371025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:56:41.371025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"118'720","reported_seq":"891","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.461342+0000","last_change":"2021-01-29T13:55:59.202455+0000","last_active":"2021-01-29T13:57:37.461342+0000","last_peered":"2021-01-29T13:57:37.461342+0000","last_clean":"2021-01-29T13:57:37.461342+0000","last_became_active":"2021-01-29T13:53:09.587252+0000","last_became_peered":"2021-01-29T13:53:09.587252+0000","last_unstale":"2021-01-29T13:57:37.461342+0000","last_undegraded":"2021-01-29T13:57:37.461342+0000","last_fullsized":"2021-01-29T13:57:37.461342+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'720","last_scrub_stamp":"2021-01-29T13:55:59.202349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:35.792587+0000","last_clean_scrub_stamp":"2021-01-29T13:55:59.202349+0000","log_size":720,"ondisk_log_size":720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2156154079,"num_objects":720,"num_object_clones":0,"num_object_copies":2880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":720,"num_write_kb":2105989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4,3],"acting":[6,2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"120'5986","reported_seq":"10744","reported_epoch":"147","state":"active+clean","last_fresh":"2021-01-29T13:57:38.478514+0000","last_change":"2021-01-29T13:57:38.478514+0000","last_active":"2021-01-29T13:57:38.478514+0000","last_peered":"2021-01-29T13:57:38.478514+0000","last_clean":"2021-01-29T13:57:38.478514+0000","last_became_active":"2021-01-29T13:57:38.475245+0000","last_became_peered":"2021-01-29T13:57:38.475245+0000","last_unstale":"2021-01-29T13:57:38.478514+0000","last_undegraded":"2021-01-29T13:57:38.478514+0000","last_fullsized":"2021-01-29T13:57:38.478514+0000","mapping_epoch":146,"log_start":"71'2904","ondisk_log_start":"71'2904","created":26,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'5955","last_scrub_stamp":"2021-01-29T13:55:51.275882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:38.811171+0000","last_clean_scrub_stamp":"2021-01-29T13:55:51.275882+0000","log_size":3082,"ondisk_log_size":3082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2947693900,"num_objects":985,"num_object_clones":0,"num_object_copies":1970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":2316,"num_read_kb":1158,"num_write":5435,"num_write_kb":1544566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1367,"num_bytes_recovered":1865847354,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":2771,"num_flush":671,"num_flush_kb":1991482,"num_evict":178,"num_evict_kb":526959,"num_promote":1172,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"171","reported_epoch":"147","state":"active+clean","last_fresh":"2021-01-29T13:57:39.387290+0000","last_change":"2021-01-29T13:57:39.387896+0000","last_active":"2021-01-29T13:57:39.387290+0000","last_peered":"2021-01-29T13:57:39.387290+0000","last_clean":"2021-01-29T13:57:39.387290+0000","last_became_active":"2021-01-29T13:53:09.377717+0000","last_became_peered":"2021-01-29T13:53:09.377717+0000","last_unstale":"2021-01-29T13:57:39.387290+0000","last_undegraded":"2021-01-29T13:57:39.387290+0000","last_fullsized":"2021-01-29T13:57:39.387290+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:55:33.131622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:55:33.131622+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"145","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.458896+0000","last_change":"2021-01-29T13:56:53.807344+0000","last_active":"2021-01-29T13:57:37.458896+0000","last_peered":"2021-01-29T13:57:37.458896+0000","last_clean":"2021-01-29T13:57:37.458896+0000","last_became_active":"2021-01-29T13:52:40.828334+0000","last_became_peered":"2021-01-29T13:52:40.828334+0000","last_unstale":"2021-01-29T13:57:37.458896+0000","last_undegraded":"2021-01-29T13:57:37.458896+0000","last_fullsized":"2021-01-29T13:57:37.458896+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:56:53.807251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:56:53.807251+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"117'6273","reported_seq":"8324","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.497913+0000","last_change":"2021-01-29T13:55:55.445357+0000","last_active":"2021-01-29T13:57:37.497913+0000","last_peered":"2021-01-29T13:57:37.497913+0000","last_clean":"2021-01-29T13:57:37.497913+0000","last_became_active":"2021-01-29T13:53:09.369472+0000","last_became_peered":"2021-01-29T13:53:09.369472+0000","last_unstale":"2021-01-29T13:57:37.497913+0000","last_undegraded":"2021-01-29T13:57:37.497913+0000","last_fullsized":"2021-01-29T13:57:37.497913+0000","mapping_epoch":42,"log_start":"71'3203","ondisk_log_start":"71'3203","created":26,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'6273","last_scrub_stamp":"2021-01-29T13:55:55.445229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:38.811171+0000","last_clean_scrub_stamp":"2021-01-29T13:55:55.445229+0000","log_size":3070,"ondisk_log_size":3070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2935339657,"num_objects":992,"num_object_clones":0,"num_object_copies":1984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":496,"num_whiteouts":0,"num_read":2414,"num_read_kb":1208,"num_write":5647,"num_write_kb":1593121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":3044,"num_flush":710,"num_flush_kb":2058922,"num_evict":219,"num_evict_kb":640347,"num_promote":1206,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"29","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.463934+0000","last_change":"2021-01-29T13:56:46.442031+0000","last_active":"2021-01-29T13:57:37.463934+0000","last_peered":"2021-01-29T13:57:37.463934+0000","last_clean":"2021-01-29T13:57:37.463934+0000","last_became_active":"2021-01-29T13:56:46.441589+0000","last_became_peered":"2021-01-29T13:56:46.441589+0000","last_unstale":"2021-01-29T13:57:37.463934+0000","last_undegraded":"2021-01-29T13:57:37.463934+0000","last_fullsized":"2021-01-29T13:57:37.463934+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-01-29T13:55:55.134762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:22.751660+0000","last_clean_scrub_stamp":"2021-01-29T13:55:55.134762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"118'6270","reported_seq":"8356","reported_epoch":"146","state":"active+clean","last_fresh":"2021-01-29T13:57:37.464735+0000","last_change":"2021-01-29T13:57:04.235378+0000","last_active":"2021-01-29T13:57:37.464735+0000","last_peered":"2021-01-29T13:57:37.464735+0000","last_clean":"2021-01-29T13:57:37.464735+0000","last_became_active":"2021-01-29T13:52:39.823642+0000","last_became_peered":"2021-01-29T13:52:39.823642+0000","last_unstale":"2021-01-29T13:57:37.464735+0000","last_undegraded":"2021-01-29T13:57:37.464735+0000","last_fullsized":"2021-01-29T13:57:37.464735+0000","mapping_epoch":26,"log_start":"71'3205","ondisk_log_start":"71'3205","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'6270","last_scrub_stamp":"2021-01-29T13:57:04.235268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-01-29T13:52:38.811171+0000","last_clean_scrub_stamp":"2021-01-29T13:57:04.235268+0000","log_size":3065,"ondisk_log_size":3065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2946120877,"num_objects":995,"num_object_clones":0,"num_object_copies":1990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":2420,"num_read_kb":1212,"num_write":5684,"num_write_kb":1601214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":3066,"num_flush":720,"num_flush_kb":2105989,"num_evict":218,"num_evict_kb":643724,"num_promote":1208,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":4,"stat_sum":{"num_bytes":11757536726,"num_objects":3958,"num_object_clones":0,"num_object_copies":7916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1960,"num_whiteouts":0,"num_read":9651,"num_read_kb":4830,"num_write":22645,"num_write_kb":6378542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1367,"num_bytes_recovered":1865847354,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":21,"num_bytes_hit_set_archive":12196,"num_flush":2862,"num_flush_kb":8348668,"num_evict":884,"num_evict_kb":2571590,"num_promote":4835,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11983056896,"data_stored":11892361364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12263,"ondisk_log_size":12263,"up":8,"acting":8,"num_store_stats":6},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":8547584878,"num_objects":2862,"num_object_clones":0,"num_object_copies":11448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":632,"num_objects_unfound":0,"num_objects_dirty":2862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2862,"num_write_kb":8348668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1840,"num_bytes_recovered":5520027622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13793206272,"data_stored":18905419776,"data_compressed":523841489,"data_compressed_allocated":5112213504,"data_compressed_original":10224427008,"omap_allocated":0,"internal_metadata":0},"log_size":2862,"ondisk_log_size":2862,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8249,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":0,"up_from":124,"seq":532575944719,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2622556,"kb_used_data":2579220,"kb_used_omap":0,"kb_used_meta":43327,"kb_avail":91749284,"statfs":{"total":96636764160,"available":93951266816,"internally_reserved":0,"allocated":2641121280,"data_stored":3110022318,"data_compressed":49186610,"data_compressed_allocated":479526912,"data_compressed_original":959053824,"omap_allocated":972,"internal_metadata":44366900},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":1.0149999999999999}]},{"osd":2,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":2.3559999999999999,"5min":2.3559999999999999,"15min":2.3559999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.93300000000000005}]},{"osd":3,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":7.8730000000000002,"5min":7.8730000000000002,"15min":7.8730000000000002},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":170.62200000000001,"5min":170.62200000000001,"15min":170.62200000000001},"last":170.62200000000001},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":2.3149999999999999,"5min":2.3149999999999999,"15min":2.3149999999999999},"last":1.258}]},{"osd":4,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.117,"15min":1.117},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":1.897,"5min":1.897,"15min":1.897},"last":1.056},{"interface":"front","average":{"1min":1.141,"5min":1.141,"15min":1.141},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.1059999999999999,"5min":2.1059999999999999,"15min":2.1059999999999999},"last":1.2370000000000001}]},{"osd":5,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":6.9889999999999999,"5min":6.9889999999999999,"15min":6.9889999999999999},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":136.00800000000001,"5min":136.00800000000001,"15min":136.00800000000001},"last":1.294},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":0.96299999999999997}]},{"osd":6,"last update":"Fri Jan 29 13:57:37 2021","interfaces":[{"interface":"back","average":{"1min":22.164000000000001,"5min":22.164000000000001,"15min":22.164000000000001},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":484.286,"5min":484.286,"15min":484.286},"last":1.1140000000000001},{"interface":"front","average":{"1min":24.145,"5min":24.145,"15min":24.145},"min":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"max":{"1min":529.46000000000004,"5min":529.46000000000004,"15min":529.46000000000004},"last":529.46000000000004}]},{"osd":7,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.274,"15min":1.274},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"max":{"1min":2.3879999999999999,"5min":2.3879999999999999,"15min":2.3879999999999999},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.264,"5min":1.264,"15min":1.264},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.349}]}]},{"osd":7,"up_from":15,"seq":64424509508,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2311472,"kb_used_data":2281064,"kb_used_omap":0,"kb_used_meta":30400,"kb_avail":92060368,"statfs":{"total":96636764160,"available":94269816832,"internally_reserved":0,"allocated":2335809536,"data_stored":3200494326,"data_compressed":88587048,"data_compressed_allocated":865042432,"data_compressed_original":1730084864,"omap_allocated":0,"internal_metadata":31129600},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.70299999999999996},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":1.004}]},{"osd":1,"last update":"Fri Jan 29 13:57:30 2021","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.242,"15min":1.46},"min":{"1min":0.64900000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.671,"5min":11.195,"15min":11.195},"last":0.91200000000000003},{"interface":"front","average":{"1min":50.901000000000003,"5min":26.797999999999998,"15min":33.463000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":951.06700000000001,"5min":951.06700000000001,"15min":951.06700000000001},"last":0.79000000000000004}]},{"osd":2,"last update":"Fri Jan 29 13:57:30 2021","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":10.314,"15min":4.5030000000000001},"min":{"1min":0.72099999999999997,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.147,"5min":512.423,"15min":512.423},"last":0.73799999999999999},{"interface":"front","average":{"1min":27.712,"5min":7.3419999999999996,"15min":3.5150000000000001},"min":{"1min":0.68600000000000005,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":505.935,"5min":505.935,"15min":505.935},"last":0.95299999999999996}]},{"osd":3,"last update":"Fri Jan 29 13:57:30 2021","interfaces":[{"interface":"back","average":{"1min":52.369999999999997,"5min":13.276,"15min":5.1639999999999997},"min":{"1min":0.749,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":977.31200000000001,"5min":977.31200000000001,"15min":977.31200000000001},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.381,"15min":1.581},"min":{"1min":1.006,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.7789999999999999,"5min":11.146000000000001,"15min":11.146000000000001},"last":1.3049999999999999}]},{"osd":4,"last update":"Fri Jan 29 13:57:17 2021","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.254,"15min":1.3340000000000001},"min":{"1min":0.79900000000000004,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.7450000000000001,"5min":5.8449999999999998,"15min":5.8449999999999998},"last":1.0469999999999999},{"interface":"front","average":{"1min":16.503,"5min":13.058,"15min":19.832000000000001},"min":{"1min":0.79700000000000004,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":309.072,"5min":391.66699999999997,"15min":391.66699999999997},"last":0.98199999999999998}]},{"osd":5,"last update":"Fri Jan 29 13:57:30 2021","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":3.3260000000000001,"15min":2.242},"min":{"1min":0.76700000000000002,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.625,"5min":175.83199999999999,"15min":175.83199999999999},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.294,"15min":1.2130000000000001},"min":{"1min":0.97099999999999997,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.8120000000000001,"5min":4.4020000000000001,"15min":4.4020000000000001},"last":1.3260000000000001}]},{"osd":6,"last update":"Fri Jan 29 13:57:30 2021","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.552,"15min":1.532},"min":{"1min":1.119,"5min":0.754,"15min":0.754},"max":{"1min":1.881,"5min":10.063000000000001,"15min":10.063000000000001},"last":1.345},{"interface":"front","average":{"1min":8.1750000000000007,"5min":9.5399999999999991,"15min":26.190000000000001},"min":{"1min":0.95299999999999996,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":132.83500000000001,"5min":623.01999999999998,"15min":623.01999999999998},"last":1.278}]}]},{"osd":2,"up_from":15,"seq":64424509507,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5660992,"kb_used_data":5606840,"kb_used_omap":0,"kb_used_meta":54144,"kb_avail":88710848,"statfs":{"total":96636764160,"available":90839908352,"internally_reserved":0,"allocated":5741404160,"data_stored":6879489840,"data_compressed":118712623,"data_compressed_allocated":1158184960,"data_compressed_original":2316369920,"omap_allocated":0,"internal_metadata":55443456},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.94299999999999995}]},{"osd":1,"last update":"Fri Jan 29 13:57:29 2021","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":17.460000000000001,"15min":6.4560000000000004},"min":{"1min":0.63200000000000001,"5min":0.499,"15min":0.499},"max":{"1min":1.3999999999999999,"5min":721.11000000000001,"15min":721.11000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.95499999999999996,"5min":21.443999999999999,"15min":7.8150000000000004},"min":{"1min":0.71899999999999997,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.3460000000000001,"5min":1000.376,"15min":1000.376},"last":0.96999999999999997}]},{"osd":3,"last update":"Fri Jan 29 13:57:29 2021","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":11.378,"15min":4.343},"min":{"1min":0.60299999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1160000000000001,"5min":501.61200000000002,"15min":501.61200000000002},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.96799999999999997,"5min":1.1819999999999999,"15min":1.0620000000000001},"min":{"1min":0.65900000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.3500000000000001,"5min":6.1870000000000003,"15min":6.1870000000000003},"last":0.90100000000000002}]},{"osd":4,"last update":"Fri Jan 29 13:57:18 2021","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.4099999999999999,"15min":1.5069999999999999},"min":{"1min":0.72699999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.6160000000000001,"5min":3.694,"15min":3.694},"last":1.1850000000000001},{"interface":"front","average":{"1min":77.519000000000005,"5min":16.635999999999999,"15min":6.4619999999999997},"min":{"1min":0.96199999999999997,"5min":0.871,"15min":0.871},"max":{"1min":874.88400000000001,"5min":874.88400000000001,"15min":874.88400000000001},"last":1.2050000000000001}]},{"osd":5,"last update":"Fri Jan 29 13:57:29 2021","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":5.7699999999999996,"15min":2.762},"min":{"1min":0.79800000000000004,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.349,"5min":183.761,"15min":183.761},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.1579999999999999,"5min":9.484,"15min":4.0389999999999997},"min":{"1min":0.88400000000000001,"5min":0.746,"15min":0.746},"max":{"1min":1.5469999999999999,"5min":521.02499999999998,"15min":521.02499999999998},"last":0.99099999999999999}]},{"osd":6,"last update":"Fri Jan 29 13:57:29 2021","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.359,"15min":1.2969999999999999},"min":{"1min":0.93500000000000005,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":1.544,"5min":3.46,"15min":3.46},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.135,"5min":1.4319999999999999,"15min":1.276},"min":{"1min":0.87,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.4590000000000001,"5min":7.0209999999999999,"15min":7.0209999999999999},"last":0.88500000000000001}]},{"osd":7,"last update":"Fri Jan 29 13:57:29 2021","interfaces":[{"interface":"back","average":{"1min":22.652000000000001,"5min":12.759,"15min":5.1719999999999997},"min":{"1min":0.98399999999999999,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":427.892,"5min":591.18899999999996,"15min":591.18899999999996},"last":1.226},{"interface":"front","average":{"1min":1.26,"5min":11.763999999999999,"15min":4.8159999999999998},"min":{"1min":0.95499999999999996,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.738,"5min":923.40099999999995,"15min":923.40099999999995},"last":1.103}]}]},{"osd":1,"up_from":15,"seq":64424509508,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2154968,"kb_used_data":2115152,"kb_used_omap":0,"kb_used_meta":39808,"kb_avail":92216872,"statfs":{"total":96636764160,"available":94430076928,"internally_reserved":0,"allocated":2165915648,"data_stored":2457997850,"data_compressed":31021897,"data_compressed_allocated":302632960,"data_compressed_original":605265920,"omap_allocated":0,"internal_metadata":40763392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:54:25 2021","interfaces":[{"interface":"back","average":{"1min":1.7629999999999999,"5min":1.0349999999999999,"15min":0.91300000000000003},"min":{"1min":0.63300000000000001,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":5.0629999999999997,"5min":5.0629999999999997,"15min":5.0629999999999997},"last":0.70799999999999996},{"interface":"front","average":{"1min":39.090000000000003,"5min":8.5060000000000002,"15min":3.4079999999999999},"min":{"1min":0.58699999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":752.73400000000004,"5min":752.73400000000004,"15min":752.73400000000004},"last":0.65200000000000002}]},{"osd":2,"last update":"Fri Jan 29 13:57:29 2021","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":16.138999999999999,"15min":27.606000000000002},"min":{"1min":0.57199999999999995,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.153,"5min":832.21600000000001,"15min":832.21600000000001},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.85299999999999998,"5min":11.237,"15min":4.4100000000000001},"min":{"1min":0.59099999999999997,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.1870000000000001,"5min":899.43399999999997,"15min":899.43399999999997},"last":0.84199999999999997}]},{"osd":3,"last update":"Fri Jan 29 13:57:29 2021","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":1.296,"15min":1.0960000000000001},"min":{"1min":0.63300000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.3169999999999999,"5min":6.0579999999999998,"15min":6.0579999999999998},"last":0.876},{"interface":"front","average":{"1min":0.84299999999999997,"5min":3.3029999999999999,"15min":9.1440000000000001},"min":{"1min":0.55100000000000005,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.1220000000000001,"5min":244.60900000000001,"15min":244.60900000000001},"last":0.82499999999999996}]},{"osd":4,"last update":"Fri Jan 29 13:57:18 2021","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.325,"15min":1.5880000000000001},"min":{"1min":0.71399999999999997,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.254,"5min":5.242,"15min":5.242},"last":0.95099999999999996},{"interface":"front","average":{"1min":1.079,"5min":1.6839999999999999,"15min":2.0870000000000002},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.2310000000000001,"5min":6.2590000000000003,"15min":6.2590000000000003},"last":1.165}]},{"osd":5,"last update":"Fri Jan 29 13:57:29 2021","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":14.631,"15min":5.681},"min":{"1min":0.68300000000000005,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.3440000000000001,"5min":701.73400000000004,"15min":701.73400000000004},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.302,"15min":1.248},"min":{"1min":0.72899999999999998,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.359,"5min":5.758,"15min":5.758},"last":1.0940000000000001}]},{"osd":6,"last update":"Fri Jan 29 13:57:30 2021","interfaces":[{"interface":"back","average":{"1min":48.951999999999998,"5min":10.974,"15min":4.5209999999999999},"min":{"1min":0.89800000000000002,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":862.01999999999998,"5min":862.01999999999998,"15min":862.01999999999998},"last":1.046},{"interface":"front","average":{"1min":1.1659999999999999,"5min":2.7810000000000001,"15min":5.8049999999999997},"min":{"1min":0.83799999999999997,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.4790000000000001,"5min":132.35300000000001,"15min":132.35300000000001},"last":1.1459999999999999}]},{"osd":7,"last update":"Fri Jan 29 13:57:29 2021","interfaces":[{"interface":"back","average":{"1min":45.155999999999999,"5min":17.600000000000001,"15min":6.7450000000000001},"min":{"1min":0.83999999999999997,"5min":0.752,"15min":0.752},"max":{"1min":749.81700000000001,"5min":749.81700000000001,"15min":749.81700000000001},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.4530000000000001,"15min":1.3979999999999999},"min":{"1min":0.85199999999999998,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":1.4790000000000001,"5min":5.9450000000000003,"15min":5.9450000000000003},"last":1.216}]}]},{"osd":3,"up_from":15,"seq":64424509508,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3443368,"kb_used_data":3391712,"kb_used_omap":0,"kb_used_meta":51648,"kb_avail":90928472,"statfs":{"total":96636764160,"available":93110755328,"internally_reserved":0,"allocated":3473113088,"data_stored":3742649427,"data_compressed":29735076,"data_compressed_allocated":290086912,"data_compressed_original":580173824,"omap_allocated":0,"internal_metadata":52887552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":29.826000000000001,"5min":29.826000000000001,"15min":29.826000000000001},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":301.69099999999997,"5min":301.69099999999997,"15min":301.69099999999997},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.613,"5min":1.613,"15min":1.613},"last":0.66800000000000004}]},{"osd":1,"last update":"Fri Jan 29 13:57:31 2021","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.96599999999999997,"15min":0.91000000000000003},"min":{"1min":0.57699999999999996,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.1339999999999999,"5min":2.5819999999999999,"15min":2.5819999999999999},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.86499999999999999,"5min":10.300000000000001,"15min":4.0890000000000004},"min":{"1min":0.59899999999999998,"5min":0.496,"15min":0.496},"max":{"1min":1.0600000000000001,"5min":977.78200000000004,"15min":977.78200000000004},"last":0.88500000000000001}]},{"osd":2,"last update":"Fri Jan 29 13:57:31 2021","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.181,"15min":1.079},"min":{"1min":0.65700000000000003,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.3919999999999999,"5min":5.7539999999999996,"15min":5.7539999999999996},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.3009999999999999,"15min":1.407},"min":{"1min":0.82999999999999996,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.2190000000000001,"5min":6.4509999999999996,"15min":6.4509999999999996},"last":1.1819999999999999}]},{"osd":4,"last update":"Fri Jan 29 13:57:15 2021","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":11.065,"15min":4.5490000000000004},"min":{"1min":0.71099999999999997,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.4890000000000001,"5min":916.09500000000003,"15min":916.09500000000003},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.131,"5min":11.628,"15min":4.859},"min":{"1min":0.81899999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.46,"5min":977.91499999999996,"15min":977.91499999999996},"last":1.0169999999999999}]},{"osd":5,"last update":"Fri Jan 29 13:57:31 2021","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":6.8769999999999998,"15min":3.1320000000000001},"min":{"1min":0.88500000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.3460000000000001,"5min":503.93000000000001,"15min":503.93000000000001},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.0269999999999999,"5min":6.7670000000000003,"15min":3.0150000000000001},"min":{"1min":0.83999999999999997,"5min":0.751,"15min":0.751},"max":{"1min":1.2569999999999999,"5min":503.74299999999999,"15min":503.74299999999999},"last":0.96199999999999997}]},{"osd":6,"last update":"Fri Jan 29 13:57:31 2021","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.4279999999999999,"15min":1.3440000000000001},"min":{"1min":1.0249999999999999,"5min":0.751,"15min":0.751},"max":{"1min":1.4710000000000001,"5min":4.7539999999999996,"15min":4.7539999999999996},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.141,"5min":10.393000000000001,"15min":34.238},"min":{"1min":0.876,"5min":0.876,"15min":0.876},"max":{"1min":1.3899999999999999,"5min":851.89099999999996,"15min":851.89099999999996},"last":1.1419999999999999}]},{"osd":7,"last update":"Fri Jan 29 13:57:31 2021","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.514,"15min":1.4910000000000001},"min":{"1min":1.0109999999999999,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.629,"5min":6.4240000000000004,"15min":6.4240000000000004},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.6830000000000001,"15min":1.917},"min":{"1min":1.038,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.6639999999999999,"5min":15.597,"15min":15.597},"last":1.246}]}]},{"osd":4,"up_from":42,"seq":180388626490,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4411548,"kb_used_data":4364052,"kb_used_omap":0,"kb_used_meta":47487,"kb_avail":89960292,"statfs":{"total":96636764160,"available":92119339008,"internally_reserved":0,"allocated":4468789248,"data_stored":5613909741,"data_compressed":118370768,"data_compressed_allocated":1155686400,"data_compressed_original":2311372800,"omap_allocated":972,"internal_metadata":48626740},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:57:36 2021","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":1.095}]},{"osd":1,"last update":"Fri Jan 29 13:57:19 2021","interfaces":[{"interface":"back","average":{"1min":9.0939999999999994,"5min":31.199000000000002,"15min":58.671999999999997},"min":{"1min":0.747,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":177.846,"5min":924.63300000000004,"15min":924.63300000000004},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.151,"5min":15.670999999999999,"15min":5.9130000000000003},"min":{"1min":0.79300000000000004,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.637,"5min":898.86900000000003,"15min":898.86900000000003},"last":1.4630000000000001}]},{"osd":2,"last update":"Fri Jan 29 13:57:19 2021","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":11.949,"15min":4.7149999999999999},"min":{"1min":0.71299999999999997,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.6020000000000001,"5min":857.78899999999999,"15min":857.78899999999999},"last":1.395},{"interface":"front","average":{"1min":0.95099999999999996,"5min":9.8019999999999996,"15min":4.0369999999999999},"min":{"1min":0.76300000000000001,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.198,"5min":779.38400000000001,"15min":779.38400000000001},"last":0.86399999999999999}]},{"osd":3,"last update":"Fri Jan 29 13:57:19 2021","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":9.625,"15min":13.391},"min":{"1min":0.69799999999999995,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.423,"5min":311.68299999999999,"15min":311.68299999999999},"last":1.032},{"interface":"front","average":{"1min":35.929000000000002,"5min":8.1660000000000004,"15min":3.5739999999999998},"min":{"1min":0.83999999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":765.98299999999995,"5min":765.98299999999995,"15min":765.98299999999995},"last":1.4319999999999999}]},{"osd":5,"last update":"Fri Jan 29 13:57:19 2021","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":3.7069999999999999,"15min":1.956},"min":{"1min":0.61899999999999999,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.238,"5min":239.99199999999999,"15min":239.99199999999999},"last":1.012},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.1639999999999999,"15min":1.1579999999999999},"min":{"1min":0.84499999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.464,"5min":3.1749999999999998,"15min":3.1749999999999998},"last":1.141}]},{"osd":6,"last update":"Fri Jan 29 13:57:19 2021","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.2370000000000001,"15min":1.2050000000000001},"min":{"1min":0.73599999999999999,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.657,"5min":5.5890000000000004,"15min":5.5890000000000004},"last":1.492},{"interface":"front","average":{"1min":1.113,"5min":1.2110000000000001,"15min":1.282},"min":{"1min":0.86599999999999999,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.6719999999999999,"5min":3.6880000000000002,"15min":3.6880000000000002},"last":1.1220000000000001}]},{"osd":7,"last update":"Fri Jan 29 13:57:19 2021","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.2869999999999999,"15min":1.2809999999999999},"min":{"1min":0.98299999999999998,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":1.5620000000000001,"5min":3.0169999999999999,"15min":3.0169999999999999},"last":1.236},{"interface":"front","average":{"1min":33.942,"5min":7.8579999999999997,"15min":3.5139999999999998},"min":{"1min":1.0109999999999999,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":719.03700000000003,"5min":719.03700000000003,"15min":719.03700000000003},"last":1.5209999999999999}]}]},{"osd":5,"up_from":15,"seq":64424509507,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":765928,"kb_used_data":751904,"kb_used_omap":0,"kb_used_meta":14016,"kb_avail":93605912,"statfs":{"total":96636764160,"available":95852453888,"internally_reserved":0,"allocated":769949696,"data_stored":1055829374,"data_compressed":29380114,"data_compressed_allocated":286253056,"data_compressed_original":572506112,"omap_allocated":0,"internal_metadata":14352384},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:57:39 2021","interfaces":[{"interface":"back","average":{"1min":24.538,"5min":24.538,"15min":24.538},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":474.77499999999998,"5min":474.77499999999998,"15min":474.77499999999998},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.98799999999999999}]},{"osd":1,"last update":"Fri Jan 29 13:57:21 2021","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.96799999999999997,"15min":0.87},"min":{"1min":0.58299999999999996,"5min":0.498,"15min":0.498},"max":{"1min":1.2350000000000001,"5min":3.1819999999999999,"15min":3.1819999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.1499999999999999,"5min":11.096,"15min":4.2889999999999997},"min":{"1min":0.84999999999999998,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.518,"5min":732.80700000000002,"15min":732.80700000000002},"last":0.91200000000000003}]},{"osd":2,"last update":"Fri Jan 29 13:57:21 2021","interfaces":[{"interface":"back","average":{"1min":12.396000000000001,"5min":10.401999999999999,"15min":4.2919999999999998},"min":{"1min":0.81599999999999995,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":225.11799999999999,"5min":649.82899999999995,"15min":649.82899999999995},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.125,"15min":1.056},"min":{"1min":0.81799999999999995,"5min":0.623,"15min":0.623},"max":{"1min":1.256,"5min":4.1799999999999997,"15min":4.1799999999999997},"last":1.2010000000000001}]},{"osd":3,"last update":"Fri Jan 29 13:57:21 2021","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":1.1160000000000001,"15min":1.1259999999999999},"min":{"1min":0.67700000000000005,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.4730000000000001,"5min":4.0890000000000004,"15min":4.0890000000000004},"last":0.86099999999999999},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.27,"15min":1.228},"min":{"1min":0.92300000000000004,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.7350000000000001,"5min":4.827,"15min":4.827},"last":0.95499999999999996}]},{"osd":4,"last update":"Fri Jan 29 13:57:18 2021","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":13.199,"15min":16.175999999999998},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.6000000000000001,"5min":514.74199999999996,"15min":514.74199999999996},"last":1.228},{"interface":"front","average":{"1min":39.335999999999999,"5min":8.8729999999999993,"15min":3.7469999999999999},"min":{"1min":0.83599999999999997,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":762.10699999999997,"5min":762.10699999999997,"15min":762.10699999999997},"last":1.1499999999999999}]},{"osd":6,"last update":"Fri Jan 29 13:57:21 2021","interfaces":[{"interface":"back","average":{"1min":37.015999999999998,"5min":16.678000000000001,"15min":6.4059999999999997},"min":{"1min":1.0900000000000001,"5min":0.622,"15min":0.622},"max":{"1min":714.803,"5min":714.803,"15min":714.803},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.4630000000000001,"15min":1.27},"min":{"1min":1.119,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.8320000000000001,"5min":4.8700000000000001,"15min":4.8700000000000001},"last":1.2669999999999999}]},{"osd":7,"last update":"Fri Jan 29 13:57:21 2021","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.423,"15min":1.5049999999999999},"min":{"1min":1.0409999999999999,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.851,"5min":9.0109999999999992,"15min":9.0109999999999992},"last":1.282},{"interface":"front","average":{"1min":1.419,"5min":1.6479999999999999,"15min":1.758},"min":{"1min":1.1319999999999999,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":1.96,"5min":9.593,"15min":9.593},"last":1.3340000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509507,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4180812,"kb_used_data":4117700,"kb_used_omap":0,"kb_used_meta":63104,"kb_avail":90191028,"statfs":{"total":96636764160,"available":92355612672,"internally_reserved":0,"allocated":4216524800,"data_stored":4766177222,"data_compressed":58393750,"data_compressed_allocated":570441728,"data_compressed_original":1140883456,"omap_allocated":0,"internal_metadata":64618496},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 29 13:57:35 2021","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.5740000000000001},"last":1.1299999999999999},{"interface":"front","average":{"1min":31.253,"5min":31.253,"15min":31.253},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":729.27700000000004,"5min":729.27700000000004,"15min":729.27700000000004},"last":0.96599999999999997}]},{"osd":1,"last update":"Fri Jan 29 13:57:26 2021","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":8.0850000000000009,"15min":26.884},"min":{"1min":0.626,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.5029999999999999,"5min":684.42700000000002,"15min":684.42700000000002},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.26,"15min":1.1619999999999999},"min":{"1min":0.69699999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.319,"5min":6.1360000000000001,"15min":6.1360000000000001},"last":1.157}]},{"osd":2,"last update":"Fri Jan 29 13:57:26 2021","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.167,"15min":1.234},"min":{"1min":0.91800000000000004,"5min":0.498,"15min":0.498},"max":{"1min":1.383,"5min":7.2329999999999997,"15min":7.2329999999999997},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.0349999999999999,"5min":8.3000000000000007,"15min":3.6179999999999999},"min":{"1min":0.60499999999999998,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.542,"5min":607.71000000000004,"15min":607.71000000000004},"last":0.86699999999999999}]},{"osd":3,"last update":"Fri Jan 29 13:57:26 2021","interfaces":[{"interface":"back","average":{"1min":29.053999999999998,"5min":6.9160000000000004,"15min":3.1040000000000001},"min":{"1min":0.77700000000000002,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":501.613,"5min":501.613,"15min":501.613},"last":1.325},{"interface":"front","average":{"1min":1.1919999999999999,"5min":6.8719999999999999,"15min":3.0489999999999999},"min":{"1min":0.88500000000000001,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.4199999999999999,"5min":480.87299999999999,"15min":480.87299999999999},"last":1.1819999999999999}]},{"osd":4,"last update":"Fri Jan 29 13:57:13 2021","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":4.6059999999999999,"15min":2.4380000000000002},"min":{"1min":0.71799999999999997,"5min":0.62,"15min":0.62},"max":{"1min":1.6439999999999999,"5min":263.584,"15min":263.584},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.238,"15min":1.3939999999999999},"min":{"1min":0.82699999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.6200000000000001,"5min":7.3449999999999998,"15min":7.3449999999999998},"last":1.423}]},{"osd":5,"last update":"Fri Jan 29 13:57:26 2021","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.27,"15min":1.3400000000000001},"min":{"1min":1.0049999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.6020000000000001,"5min":4.0069999999999997,"15min":4.0069999999999997},"last":1.27},{"interface":"front","average":{"1min":1.25,"5min":1.569,"15min":1.492},"min":{"1min":1.012,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":1.5409999999999999,"5min":7.3140000000000001,"15min":7.3140000000000001},"last":1.5940000000000001}]},{"osd":7,"last update":"Fri Jan 29 13:57:26 2021","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.4159999999999999,"15min":1.3380000000000001},"min":{"1min":0.91100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.581,"5min":10.795,"15min":10.795},"last":1.655},{"interface":"front","average":{"1min":1.278,"5min":1.365,"15min":1.3839999999999999},"min":{"1min":0.99299999999999999,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.722,"5min":4.0579999999999998,"15min":4.0579999999999998},"last":1.383}]}]}]}} , 2021-01-29T13:57:47.178 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-01-29T13:57:47.465 INFO:teuthology.orchestra.run.smithi183.stdout: 2021-01-30T06:14:58.019 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T13:58:10.671 INFO:teuthology.orchestra.run.smithi183.stdout:} , 2021-01-29T13:58:10.682 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 , instructed osd(s) 2 to deep-scrub 2021-01-30T06:14:58.020 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 179, in _raise_for_status , raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 10000 --objects 6600 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 1200 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2021-01-30T06:14:58.021 WARNING:scrape:Ignoring malparsed backtrace: 2021-01-29T13:59:00.217 DEBUG:teuthology.orchestra.run.smithi183:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- , 2021-01-29T14:01:20.328 INFO:tasks.ceph:Archiving logs... , 2021-01-29T14:01:20.329 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/log/ceph to /home/teuthworker/archive/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840036/remote/smithi183/log 2021-01-30T06:14:58.022 WARNING:scrape:Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 368, in archive , yield 2021-01-30T06:14:58.218 INFO:scrape:Found 9 distinct failure reasons 2021-01-30T06:14:58.220 INFO:scrape:Failure: Command failed (workunit test mon/misc.sh) on smithi007 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=27dc1eb32728beb0c69aa65131bc609ef680c0aa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/misc.sh' 2021-01-30T06:14:58.221 INFO:scrape:2 jobs: ['5840169', '5840388'] 2021-01-30T06:14:58.221 INFO:scrape:suites intersection: ['workloads/mon}'] 2021-01-30T06:14:58.222 INFO:scrape:suites union: ['rados/standalone/{mon_election/classic', 'rados/standalone/{mon_election/connectivity', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/mon}'] 2021-01-30T06:14:58.223 INFO:scrape: 2021-01-30T06:14:58.223 INFO:scrape:Failure: Exiting scrub checking -- not all pgs scrubbed. 2021-01-30T06:14:58.224 INFO:scrape:2 jobs: ['5840339', '5840141'] 2021-01-30T06:14:58.225 INFO:scrape:suites intersection: ['1-pg-log-overrides/short_pg_log', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'thrashers/morepggrow', 'thrashosds-health'] 2021-01-30T06:14:58.226 INFO:scrape:suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/cache-pool-snaps-readproxy}', 'workloads/cache-snaps-balanced}'] 2021-01-30T06:14:58.227 INFO:scrape: 2021-01-30T06:14:58.227 INFO:scrape:Dead: b'2021-01-30T03:47:08.841 DEBUG:teuthology.exit:Finished running handlers' 2021-01-30T06:14:58.228 INFO:scrape:8 jobs: ['5840201', '5840076', '5840359', '5840022', '5840182', '5840158', '5839984', '5840379'] 2021-01-30T06:14:58.228 INFO:scrape:suites intersection: [] 2021-01-30T06:14:58.229 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-async-recovery}', 'backoff/normal', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'fixed-2', 'mode/packaged', 'mode/root', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack', 'openstack}', 'rados', 'rados/cephadm/with-work/{distro/ubuntu_20.04', 'rados/perf/{ceph', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'scheduler/dmclock_default_shards', 'scheduler/wpq_default_shards', 'settings/optimized', 'start', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'thrashers/careful', 'thrashers/mapgap', 'thrashosds-health', 'ubuntu_latest', 'workloads/cosbench_64K_read_write}', 'workloads/cosbench_64K_write}', 'workloads/radosbench-high-concurrency}', 'workloads/radosbench_omap_write}'] 2021-01-30T06:14:58.229 INFO:scrape: 2021-01-30T06:14:58.230 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2021-01-30T06:14:58.230 INFO:scrape:3 jobs: ['5840243', '5840187', '5840036'] 2021-01-30T06:14:58.231 INFO:scrape:suites intersection: ['mon_election/connectivity', 'rados'] 2021-01-30T06:14:58.231 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-partial-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/many', 'msgr-failures/osd-delay', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/cache-agent-big}', 'workloads/redirect_set_object}'] 2021-01-30T06:14:58.232 INFO:scrape: 2021-01-30T06:14:58.232 INFO:scrape:Failure: Command failed on smithi013 with status 2: 'git archive --remote=git://git.ceph.com/ceph-ci.git pacific src/cephadm/cephadm | tar -xO src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm' 2021-01-30T06:14:58.233 INFO:scrape:2 jobs: ['5840045', '5840266'] 2021-01-30T06:14:58.233 INFO:scrape:suites intersection: ['1-tasks', 'rados/upgrade/pacific-x/parallel/{0-start', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2021-01-30T06:14:58.234 INFO:scrape:suites union: ['1-tasks', 'distro1$/{centos_8.0}', 'distro1$/{ubuntu_18.04}', 'mon_election/classic', 'mon_election/connectivity', 'rados/upgrade/pacific-x/parallel/{0-start', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2021-01-30T06:14:58.234 INFO:scrape: 2021-01-30T06:14:58.234 INFO:scrape:Failure: SSH connection to smithi107 was lost: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 74b1d184-625c-11eb-8fa5-001a4aab830c --force' 2021-01-30T06:14:58.235 INFO:scrape:1 jobs: ['5840332'] 2021-01-30T06:14:58.235 INFO:scrape:suites: ['2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro$/{ubuntu_20.04}', 'fixed-2', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start'] 2021-01-30T06:14:58.236 INFO:scrape: 2021-01-30T06:14:58.236 INFO:scrape:Failure: [Errno 113] No route to host 2021-01-30T06:14:58.237 INFO:scrape:1 jobs: ['5840171'] 2021-01-30T06:14:58.237 INFO:scrape:suites: ['2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro$/{ubuntu_20.04}', 'fixed-2', 'mon_election/classic}', 'rados/cephadm/upgrade/{1-start'] 2021-01-30T06:14:58.238 INFO:scrape: 2021-01-30T06:14:58.238 INFO:scrape:Failure: "2021-01-29T13:35:11.838843+0000 mon.a (mon.0) 11 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log 2021-01-30T06:14:58.239 INFO:scrape:1 jobs: ['5840011'] 2021-01-30T06:14:58.239 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'msgr/async-v2only', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/small-objects}'] 2021-01-30T06:14:58.240 INFO:scrape: 2021-01-30T06:14:58.240 INFO:scrape:Failure: Command failed (workunit test cephtool/test.sh) on smithi003 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=27dc1eb32728beb0c69aa65131bc609ef680c0aa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' 2021-01-30T06:14:58.240 INFO:scrape:1 jobs: ['5840263'] 2021-01-30T06:14:58.241 INFO:scrape:suites: ['mon_election/classic', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-bitmap', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{centos_8}}'] 2021-01-30T06:14:58.241 INFO:scrape: 2021-01-30T06:14:58.257 INFO:teuthology.results:Sending results to tchaikov@gmail.com: Test Run: kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/ failed: 12 dead: 9 running: 0 waiting: 0 queued: 0 passed: 418 Fail ================================================================= [5840011] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:17:19 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840011/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840011/ "2021-01-29T13:35:11.838843+0000 mon.a (mon.0) 11 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log [5840036] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/cache-agent-big} ----------------------------------------------------------------- time: 00:18:25 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840036/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840036/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 10000 --objects 6600 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 1200 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' [5840045] rados/upgrade/pacific-x/parallel/{0-start 1-tasks distro1$/{centos_8.0} mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:07:57 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840045/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840045/ sentry: https://sentry.ceph.com/organizations/ceph/?query=089fba1d3ad4456190d481bebc06095a Command failed on smithi013 with status 2: 'git archive --remote=git://git.ceph.com/ceph-ci.git pacific src/cephadm/cephadm | tar -xO src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm' [5840141] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps-readproxy} ----------------------------------------------------------------- time: 00:42:52 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840141/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840141/ Exiting scrub checking -- not all pgs scrubbed. [5840169] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/mon} ----------------------------------------------------------------- time: 00:12:07 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840169/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840169/ sentry: https://sentry.ceph.com/organizations/ceph/?query=08b6d8e0e6fc4a8496810fbed85aca07 Command failed (workunit test mon/misc.sh) on smithi007 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=27dc1eb32728beb0c69aa65131bc609ef680c0aa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/misc.sh' [5840171] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{ubuntu_20.04} fixed-2 mon_election/classic} ----------------------------------------------------------------- time: 00:19:46 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840171/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840171/ [Errno 113] No route to host [5840187] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/redirect_set_object} ----------------------------------------------------------------- time: 00:13:49 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840187/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840187/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_redirect --low_tier_pool low_tier --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max- seconds 0 --op read 100 --op write 50 --op delete 10 --op copy_from 100 --op write_excl 50 --pool unique_pool_0' [5840243] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:14:01 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840243/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840243/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' [5840263] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:28:20 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840263/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840263/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d982fb73bbd342ffb47ef269ab26d14e Command failed (workunit test cephtool/test.sh) on smithi003 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=27dc1eb32728beb0c69aa65131bc609ef680c0aa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' [5840266] rados/upgrade/pacific-x/parallel/{0-start 1-tasks distro1$/{ubuntu_18.04} mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:04:11 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840266/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840266/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a02d5b6aafa449d18e968e9193447e6b Command failed on smithi043 with status 2: 'git archive --remote=git://git.ceph.com/ceph-ci.git pacific src/cephadm/cephadm | tar -xO src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm' [5840339] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache-snaps-balanced} ----------------------------------------------------------------- time: 00:49:25 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840339/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840339/ Exiting scrub checking -- not all pgs scrubbed. [5840388] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/mon} ----------------------------------------------------------------- time: 00:07:55 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840388/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840388/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1178259313e94ee2a4b52f18787d4b7f Command failed (workunit test mon/misc.sh) on smithi013 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=27dc1eb32728beb0c69aa65131bc609ef680c0aa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/misc.sh' Dead ================================================================= [5839984] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/radosbench-high-concurrency} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5839984/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5839984/ [5840022] rados/cephadm/with-work/{distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840022/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840022/ [5840076] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840076/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840076/ [5840158] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/cosbench_64K_read_write} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840158/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840158/ [5840182] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/cosbench_64K_write} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840182/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840182/ [5840201] rados/cephadm/with-work/{distro/ubuntu_20.04 fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_python} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840201/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840201/ [5840332] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{ubuntu_20.04} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:50:54 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840332/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840332/ SSH connection to smithi107 was lost: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 74b1d184-625c-11eb-8fa5-001a4aab830c --force' [5840359] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/radosbench-high-concurrency} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840359/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840359/ [5840379] rados/cephadm/with-work/{distro/ubuntu_20.04 fixed-2 mode/root mon_election/classic msgr/async-v2only start tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840379/ log: http://qa-proxy.ceph.com/teuthology/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840379/ Pass ================================================================= [5839977] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:46 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5839977/ [5839978] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/rados_api_tests} time: 00:26:59 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5839978/ [5839979] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:19:27 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5839979/ [5839980] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_cephadm} time: 00:14:53 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5839980/ [5839981] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:09:32 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5839981/ [5839982] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/osd} time: 03:14:43 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5839982/ [5839983] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-stupid supported-random-distro$/{rhel_8} tasks/workunits} time: 00:10:27 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5839983/ [5839985] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_python} time: 00:21:53 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5839985/ [5839986] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/radosbench} time: 02:28:48 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5839986/ [5839987] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/connectivity start} time: 00:12:54 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5839987/ [5839988] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/redirect} time: 00:11:31 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5839988/ [5839989] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:15:13 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5839989/ [5839990] rados/singleton/{all/mon-auth-caps mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:10:14 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5839990/ [5839991] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:27 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5839991/ [5839992] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados tasks/rados_cls_all validater/lockdep} time: 00:19:07 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5839992/ [5839993] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/octopus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/rbd_cls} time: 00:21:23 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5839993/ [5839994] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-stupid supported-random-distro$/{centos_8} tasks/dashboard} time: 01:10:52 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5839994/ [5839995] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/redirect_promote_tests} time: 00:14:20 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5839995/ [5839996] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:08:53 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5839996/ [5839997] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{centos_8.0} fixed-2 mon_election/connectivity} time: 00:18:14 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5839997/ [5839998] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/redirect_set_object} time: 00:14:00 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5839998/ [5839999] rados/multimon/{clusters/9 mon_election/classic msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} tasks/mon_recovery} time: 00:16:25 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5839999/ [5840000] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:31:32 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840000/ [5840001] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/set-chunks-read} time: 00:16:11 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840001/ [5840002] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_cephadm_repos} time: 00:07:01 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840002/ [5840003] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:09:26 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840003/ [5840004] rados/singleton/{all/mon-config-key-caps mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:11:07 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840004/ [5840005] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} tasks/rados_api_tests} time: 00:27:15 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840005/ [5840006] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/small-objects-balanced} time: 00:20:19 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840006/ [5840007] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/pacific backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:37:52 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840007/ [5840008] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/small-objects-localized} time: 00:22:08 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840008/ [5840009] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/scrub} time: 01:50:12 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840009/ [5840010] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/classic start} time: 00:13:16 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840010/ [5840012] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/pool-create-delete} time: 00:12:47 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840012/ [5840013] rados/singleton/{all/mon-config-keys mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:11:32 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840013/ [5840014] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:36:37 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840014/ [5840015] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:24 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840015/ [5840016] rados/cephadm/smoke-roleless/{distro/ubuntu_20.04 start} time: 00:12:44 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840016/ [5840017] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:24:03 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840017/ [5840018] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:51 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840018/ [5840019] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:06:55 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840019/ [5840020] rados/objectstore/{backends/filestore-idempotent supported-random-distro$/{rhel_8}} time: 02:18:42 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840020/ [5840021] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-balanced} time: 00:27:29 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840021/ [5840023] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:08:03 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840023/ [5840024] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/snaps-few-objects-localized} time: 00:23:56 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840024/ [5840025] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/filestore-xfs supported-random-distro$/{centos_8} tasks/crash} time: 00:11:35 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840025/ [5840026] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_orch_cli} time: 00:18:37 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840026/ [5840027] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/snaps-few-objects} time: 00:29:27 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840027/ [5840028] rados/singleton/{all/mon-config mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:06:34 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840028/ [5840029] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:10:56 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840029/ [5840030] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/careful thrashosds-health workloads/test_rbd_api} time: 00:19:03 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840030/ [5840031] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/write_fadvise_dontneed} time: 00:14:19 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840031/ [5840032] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} time: 00:33:21 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840032/ [5840033] rados/cephadm/smoke/{distro/ubuntu_20.04 fixed-2 mon_election/connectivity start} time: 00:14:15 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840033/ [5840034] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:32:17 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840034/ [5840035] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/mon_recovery validater/valgrind} time: 00:29:09 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840035/ [5840037] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/default thrashosds-health workloads/cache-snaps} time: 00:30:12 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840037/ [5840038] rados/singleton/{all/osd-backfill mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} time: 00:28:54 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840038/ [5840039] rados/multimon/{clusters/21 mon_election/connectivity msgr-failures/few msgr/async no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:05:30 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840039/ [5840040] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/rados_cls_all} time: 00:16:37 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840040/ [5840041] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:08:20 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840041/ [5840042] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:31:42 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840042/ [5840043] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:14:45 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840043/ [5840044] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/crush} time: 00:08:58 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840044/ [5840046] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-agent-small} time: 00:11:33 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840046/ [5840047] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/root mon_election/classic msgr/async-v2only start tasks/rados_python} time: 00:23:36 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840047/ [5840048] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:22:42 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840048/ [5840049] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:07:15 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840049/ [5840050] rados/cephadm/dashboard/{distro/centos_latest task/test_e2e} time: 00:19:13 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840050/ [5840051] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache-pool-snaps} time: 00:30:02 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840051/ [5840052] rados/singleton/{all/osd-recovery-incomplete mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:27:12 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840052/ [5840053] rados/cephadm/orchestrator_cli/{2-node-mgr orchestrator_cli supported-random-distro$/{centos_8}} time: 00:13:11 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840053/ [5840054] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/cache-snaps-balanced} time: 00:22:01 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840054/ [5840055] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:10:41 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840055/ [5840056] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:04 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840056/ [5840057] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/force-sync-many workloads/rados_5925} time: 00:10:45 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840057/ [5840058] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:26:48 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840058/ [5840059] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/classic start} time: 00:14:05 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840059/ [5840060] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/cache-snaps} time: 00:22:21 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840060/ [5840061] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:24:00 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840061/ [5840062] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:18:12 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840062/ [5840063] rados/cephadm/smoke-roleless/{distro/centos_8.0 start} time: 00:13:28 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840063/ [5840064] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache} time: 00:08:00 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840064/ [5840065] rados/singleton/{all/osd-recovery mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:14:30 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840065/ [5840066] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/failover} time: 00:14:21 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840066/ [5840067] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8} tasks/dashboard} time: 01:10:52 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840067/ [5840068] rados/objectstore/{backends/fusestore supported-random-distro$/{ubuntu_latest}} time: 00:04:53 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840068/ [5840069] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/pool-snaps-few-objects} time: 00:28:44 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840069/ [5840070] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{centos_8.0} fixed-2 mon_election/classic} time: 00:19:41 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840070/ [5840071] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:06:21 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840071/ [5840072] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/rados_api_tests} time: 00:24:25 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840072/ [5840073] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_adoption} time: 00:09:21 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840073/ [5840074] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/erasure-code} time: 00:37:58 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840074/ [5840075] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/radosbench-high-concurrency} time: 00:52:57 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840075/ [5840077] rados/singleton/{all/peer mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} time: 00:05:38 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840077/ [5840078] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/mapgap thrashosds-health workloads/radosbench} time: 01:25:17 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840078/ [5840079] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:12:28 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840079/ [5840080] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/lockdep} time: 00:26:37 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840080/ [5840081] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} tasks/rados_python} time: 00:15:57 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840081/ [5840082] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/radosbench} time: 01:32:13 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840082/ [5840083] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:09:37 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840083/ [5840084] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/connectivity start} time: 00:13:40 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840084/ [5840085] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/redirect} time: 00:18:03 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840085/ [5840086] rados/multimon/{clusters/3 mon_election/classic msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} tasks/mon_clock_with_skews} time: 00:08:08 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840086/ [5840087] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_cephadm} time: 00:14:21 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840087/ [5840088] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/redirect_promote_tests} time: 00:13:53 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840088/ [5840089] rados/singleton/{all/pg-autoscaler-progress-off mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:08:40 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840089/ [5840090] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-snappy supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:08:14 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840090/ [5840091] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/morepggrow thrashosds-health workloads/rbd_cls} time: 00:19:50 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840091/ [5840092] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/redirect_set_object} time: 00:10:24 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840092/ [5840093] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:08:01 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840093/ [5840094] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/set-chunks-read} time: 00:14:19 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840094/ [5840095] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/packaged mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:33:21 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840095/ [5840096] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:57 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840096/ [5840097] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/small-objects-balanced} time: 00:21:37 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840097/ [5840098] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/many workloads/rados_api_tests} time: 00:34:38 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840098/ [5840099] rados/singleton/{all/pg-autoscaler mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:12:23 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840099/ [5840100] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose objectstore/filestore-xfs rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:21:17 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840100/ [5840101] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/classic start} time: 00:13:07 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840101/ [5840102] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_fio} time: 00:13:19 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840102/ [5840103] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:50 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840103/ [5840104] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/small-objects-localized} time: 00:22:02 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840104/ [5840105] rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/mgr} time: 00:11:57 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840105/ [5840106] rados/cephadm/smoke-roleless/{distro/centos_latest start} time: 00:12:25 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840106/ [5840107] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} tasks/insights} time: 00:14:08 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840107/ [5840108] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:05:04 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840108/ [5840109] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/small-objects} time: 00:24:17 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840109/ [5840110] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/none thrashosds-health workloads/snaps-few-objects} time: 00:29:29 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840110/ [5840111] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-balanced} time: 00:30:10 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840111/ [5840112] rados/singleton/{all/pg-removal-interruption mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:08:57 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840112/ [5840113] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{centos_8.0} fixed-2 mon_election/connectivity} time: 00:18:58 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840113/ [5840114] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} tasks/rados_stress_watch} time: 00:17:54 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840114/ [5840115] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zlib supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:08:53 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840115/ [5840116] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{centos_8}} time: 00:13:27 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840116/ [5840117] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} time: 00:21:45 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840117/ [5840118] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-localized} time: 00:31:43 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840118/ [5840119] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_cephadm_repos} time: 00:06:35 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840119/ [5840120] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:08:54 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840120/ [5840121] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/snaps-few-objects} time: 00:24:26 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840121/ [5840122] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:27:26 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840122/ [5840123] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados tasks/rados_cls_all validater/valgrind} time: 02:10:33 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840123/ [5840124] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-overwrites} time: 00:23:33 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840124/ [5840125] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/connectivity start} time: 00:13:58 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840125/ [5840126] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/write_fadvise_dontneed} time: 00:15:48 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840126/ [5840127] rados/singleton/{all/radostool mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:11:16 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840127/ [5840128] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/few msgr/async-v2only no_pools objectstore/filestore-xfs rados supported-random-distro$/{centos_8} tasks/mon_recovery} time: 00:11:36 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840128/ [5840129] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/admin_socket_objecter_requests} time: 00:33:01 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840129/ [5840130] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/root mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:23:12 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840130/ [5840131] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:08:16 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840131/ [5840132] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:07:26 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840132/ [5840133] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/cache-agent-big} time: 00:25:14 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840133/ [5840134] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} time: 00:20:32 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840134/ [5840135] rados/standalone/{mon_election/classic supported-random-distro$/{rhel_8} workloads/misc} time: 00:31:10 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840135/ [5840136] rados/singleton/{all/random-eio mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8}} time: 00:34:48 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840136/ [5840137] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache-agent-small} time: 00:12:57 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840137/ [5840138] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:25 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840138/ [5840139] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_orch_cli} time: 00:18:54 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840139/ [5840140] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} tasks/dashboard} time: 01:09:38 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840140/ [5840142] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/one workloads/rados_mon_osdmap_prune} time: 00:51:57 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840142/ [5840143] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-radosbench} time: 04:16:25 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840143/ [5840144] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/careful thrashosds-health workloads/cache-snaps} time: 00:30:33 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840144/ [5840145] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-pool-snaps} time: 00:17:38 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840145/ [5840146] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:10:12 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840146/ [5840147] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:25 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840147/ [5840148] rados/cephadm/smoke/{distro/ubuntu_20.04 fixed-2 mon_election/classic start} time: 00:14:34 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840148/ [5840149] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} tasks/module_selftest} time: 00:26:07 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840149/ [5840150] rados/singleton/{all/rebuild-mondb mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}} time: 00:15:00 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840150/ [5840151] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache-snaps-balanced} time: 00:26:41 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840151/ [5840152] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} tasks/rados_striper} time: 00:09:47 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840152/ [5840153] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:32:30 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840153/ [5840154] rados/cephadm/smoke-roleless/{distro/rhel_latest start} time: 00:12:04 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840154/ [5840155] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/cache-snaps} time: 00:25:12 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840155/ [5840156] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} time: 00:33:32 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840156/ [5840157] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache} time: 00:09:24 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840157/ [5840159] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:11:14 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840159/ [5840160] rados/singleton/{all/recovery-preemption mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:16:37 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840160/ [5840161] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/pool-snaps-few-objects} time: 00:29:08 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840161/ [5840162] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/default thrashosds-health workloads/radosbench} time: 01:56:08 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840162/ [5840163] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:10:05 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840163/ [5840164] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{centos_8}} time: 00:25:10 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840164/ [5840165] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:13:04 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840165/ [5840166] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_cls_all validater/lockdep} time: 00:20:59 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840166/ [5840167] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/rados_api_tests} time: 00:28:55 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840167/ [5840168] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/connectivity start} time: 00:14:08 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840168/ [5840170] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/radosbench-high-concurrency} time: 01:11:05 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840170/ [5840172] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:10:12 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840172/ [5840173] rados/multimon/{clusters/9 mon_election/classic msgr-failures/many msgr/async no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews} time: 00:08:31 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840173/ [5840174] rados/singleton/{all/resolve_stuck_peering mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} time: 00:10:18 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840174/ [5840175] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/radosbench} time: 01:15:48 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840175/ [5840176] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_adoption} time: 00:09:30 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840176/ [5840177] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/redirect} time: 00:11:22 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840177/ [5840178] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/filestore-xfs rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:25 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840178/ [5840179] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/pacific backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/mapgap thrashosds-health workloads/rbd_cls} time: 00:22:10 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840179/ [5840180] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/redirect_promote_tests} time: 00:17:25 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840180/ [5840181] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:15:00 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840181/ [5840183] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/sync-many workloads/rados_mon_workunits} time: 00:26:23 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840183/ [5840184] rados/singleton/{all/test-crash mon_election/classic msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8}} time: 00:10:45 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840184/ [5840185] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-small-objects-balanced} time: 00:21:34 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840185/ [5840186] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/classic start} time: 00:13:25 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840186/ [5840188] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:29:28 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840188/ [5840189] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} tasks/rados_workunit_loadgen_big} time: 00:32:25 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840189/ [5840190] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} tasks/dashboard} time: 01:12:21 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840190/ [5840191] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:58 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840191/ [5840192] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/set-chunks-read} time: 00:16:55 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840192/ [5840193] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04 start} time: 00:11:40 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840193/ [5840194] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zlib supported-random-distro$/{rhel_8} tasks/progress} time: 00:18:30 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840194/ [5840195] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/small-objects-balanced} time: 00:20:46 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840195/ [5840196] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_cephadm} time: 00:14:31 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840196/ [5840197] rados/singleton/{all/test_envlibrados_for_rocksdb mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} time: 00:16:58 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840197/ [5840198] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:21:21 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840198/ [5840199] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/small-objects-localized} time: 00:23:25 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840199/ [5840200] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/osd} time: 03:18:17 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840200/ [5840202] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/small-objects} time: 00:17:47 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840202/ [5840203] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:37:58 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840203/ [5840204] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} time: 00:31:43 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840204/ [5840205] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:30:11 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840205/ [5840206] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados tasks/mon_recovery validater/valgrind} time: 00:27:40 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840206/ [5840207] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/connectivity start} time: 00:12:43 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840207/ [5840208] rados/singleton/{all/thrash-backfill-full mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 01:31:56 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840208/ [5840209] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:15:15 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840209/ [5840210] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-localized} time: 00:24:35 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840210/ [5840211] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:10:14 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840211/ [5840212] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-stupid supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:11:57 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840212/ [5840213] rados/objectstore/{backends/objectstore supported-random-distro$/{centos_8}} time: 05:03:07 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840213/ [5840214] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/none thrashosds-health workloads/test_rbd_api} time: 00:14:45 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840214/ [5840215] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:31:29 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840215/ [5840216] rados/multimon/{clusters/21 mon_election/connectivity msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:05:24 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840216/ [5840217] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{centos_latest} fixed-2 mon_election/connectivity} time: 00:18:05 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840217/ [5840218] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/write_fadvise_dontneed} time: 00:13:15 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840218/ [5840219] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:06 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840219/ [5840220] rados/singleton/{all/thrash-eio mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:31:15 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840220/ [5840221] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/admin_socket_objecter_requests} time: 00:32:04 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840221/ [5840222] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_cephadm_repos} time: 00:06:41 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840222/ [5840223] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:10:10 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840223/ [5840224] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} tasks/rados_workunit_loadgen_mix} time: 00:21:37 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840224/ [5840225] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/cache-agent-big} time: 00:26:39 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840225/ [5840226] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/snaps-few-objects} time: 00:25:54 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840226/ [5840227] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-fast-read} time: 00:21:16 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840227/ [5840228] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/classic start} time: 00:12:47 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840228/ [5840229] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/scrub} time: 01:49:44 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840229/ [5840230] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache-agent-small} time: 00:12:24 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840230/ [5840231] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} tasks/prometheus} time: 00:11:57 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840231/ [5840232] rados/cephadm/smoke-roleless/{distro/ubuntu_20.04 start} time: 00:12:26 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840232/ [5840233] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:24:56 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840233/ [5840234] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} time: 00:21:31 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840234/ [5840235] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:26:48 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840235/ [5840236] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:28:54 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840236/ [5840237] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/root mon_election/classic msgr/async-v1only start tasks/rados_api_tests} time: 00:34:39 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840237/ [5840238] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/filestore-xfs supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:10:32 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840238/ [5840239] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps} time: 00:27:54 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840239/ [5840240] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:10:42 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840240/ [5840241] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/pggrow thrashosds-health workloads/cache-snaps} time: 00:30:46 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840241/ [5840242] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/cache-snaps-balanced} time: 00:19:37 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840242/ [5840244] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_orch_cli} time: 00:18:44 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840244/ [5840245] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache-snaps} time: 00:26:53 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840245/ [5840246] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:10:35 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840246/ [5840247] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:25:31 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840247/ [5840248] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:08:56 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840248/ [5840249] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} time: 00:25:52 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840249/ [5840250] rados/cephadm/smoke/{distro/ubuntu_20.04 fixed-2 mon_election/connectivity start} time: 00:14:06 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840250/ [5840251] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/cache} time: 00:14:14 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840251/ [5840252] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/pool-snaps-few-objects} time: 00:30:29 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840252/ [5840253] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/careful thrashosds-health workloads/radosbench} time: 01:28:30 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840253/ [5840254] rados/singleton/{all/watch-notify-same-primary mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} time: 00:09:50 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840254/ [5840255] rados/multimon/{clusters/3 mon_election/classic msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} tasks/mon_recovery} time: 00:10:48 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840255/ [5840256] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/rados_api_tests} time: 00:24:46 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840256/ [5840257] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:26 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840257/ [5840258] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rados_python} time: 00:22:02 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840258/ [5840259] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} tasks/rados_workunit_loadgen_mostlyread} time: 00:33:03 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840259/ [5840260] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:07:31 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840260/ [5840261] rados/objectstore/{backends/alloc-hint supported-random-distro$/{centos_8}} time: 00:09:30 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840261/ [5840262] rados/rest/{mgr-restful supported-random-distro$/{ubuntu_latest}} time: 00:07:22 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840262/ [5840264] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:16:23 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840264/ [5840265] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/crush} time: 00:13:13 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840265/ [5840267] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:21:15 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840267/ [5840268] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/radosbench-high-concurrency} time: 01:06:54 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840268/ [5840269] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/default thrashosds-health workloads/rbd_cls} time: 00:21:04 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840269/ [5840270] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:10:01 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840270/ [5840271] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/radosbench} time: 02:00:47 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840271/ [5840272] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/pool-create-delete} time: 00:12:30 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840272/ [5840273] rados/singleton/{all/admin-socket mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:09:17 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840273/ [5840274] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:27:09 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840274/ [5840275] rados/cephadm/dashboard/{distro/centos_latest task/test_e2e} time: 00:19:31 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840275/ [5840276] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/redirect} time: 00:14:34 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840276/ [5840277] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} tasks/workunits} time: 00:06:46 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840277/ [5840278] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:11:00 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840278/ [5840279] rados/cephadm/orchestrator_cli/{2-node-mgr orchestrator_cli supported-random-distro$/{ubuntu_latest}} time: 00:10:16 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840279/ [5840280] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/redirect_promote_tests} time: 00:16:43 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840280/ [5840281] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:32:53 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840281/ [5840282] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/classic start} time: 00:14:04 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840282/ [5840283] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/redirect_set_object} time: 00:12:37 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840283/ [5840284] rados/singleton/{all/deduptool mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:06:00 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840284/ [5840285] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/set-chunks-read} time: 00:14:53 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840285/ [5840286] rados/cephadm/smoke-roleless/{distro/centos_8.0 start} time: 00:13:00 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840286/ [5840287] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:10:28 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840287/ [5840288] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:06:22 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840288/ [5840289] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/small-objects-balanced} time: 00:19:15 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840289/ [5840290] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{ubuntu_18.04} fixed-2 mon_election/classic} time: 00:17:42 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840290/ [5840291] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/small-objects-localized} time: 00:16:44 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840291/ [5840292] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/erasure-code} time: 00:37:41 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840292/ [5840293] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_adoption} time: 00:09:15 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840293/ [5840294] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:08:32 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840294/ [5840295] rados/singleton/{all/divergent_priors mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8}} time: 00:09:52 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840295/ [5840296] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:29:57 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840296/ [5840297] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados tasks/rados_cls_all validater/valgrind} time: 02:10:40 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840297/ [5840298] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/small-objects} time: 00:21:50 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840298/ [5840299] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v2only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/mapgap thrashosds-health workloads/snaps-few-objects} time: 00:41:17 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840299/ [5840300] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} tasks/readwrite} time: 00:12:46 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840300/ [5840301] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:44 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840301/ [5840302] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/snaps-few-objects-balanced} time: 00:34:24 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840302/ [5840303] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:10:10 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840303/ [5840304] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/few msgr/async no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} tasks/mon_clock_no_skews} time: 00:08:45 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840304/ [5840305] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/connectivity start} time: 00:13:54 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840305/ [5840306] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/snaps-few-objects-localized} time: 00:31:25 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840306/ [5840307] rados/singleton/{all/divergent_priors2 mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 00:09:43 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840307/ [5840308] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_cephadm} time: 00:15:51 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840308/ [5840309] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/snaps-few-objects} time: 00:25:53 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840309/ [5840310] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-snappy supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:09:52 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840310/ [5840311] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{ubuntu_latest}} time: 00:09:34 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840311/ [5840312] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:31:10 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840312/ [5840313] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_5925} time: 00:06:31 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840313/ [5840314] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-small-objects} time: 00:24:44 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840314/ [5840315] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:34:26 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840315/ [5840316] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/write_fadvise_dontneed} time: 00:11:56 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840316/ [5840317] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:05:00 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840317/ [5840318] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} tasks/crash} time: 00:11:58 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840318/ [5840319] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/admin_socket_objecter_requests} time: 00:24:53 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840319/ [5840320] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/morepggrow thrashosds-health workloads/test_rbd_api} time: 00:20:19 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840320/ [5840321] rados/singleton/{all/dump-stuck mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:10:15 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840321/ [5840322] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:24:14 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840322/ [5840323] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache-agent-big} time: 00:18:51 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840323/ [5840324] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:08:57 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840324/ [5840325] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/classic start} time: 00:13:15 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840325/ [5840326] rados/standalone/{mon_election/classic supported-random-distro$/{rhel_8} workloads/mgr} time: 00:12:43 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840326/ [5840327] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/cache-agent-small} time: 00:13:42 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840327/ [5840328] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:08:56 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840328/ [5840329] rados/cephadm/smoke-roleless/{distro/centos_latest start} time: 00:12:51 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840329/ [5840330] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:24:47 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840330/ [5840331] rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:56:17 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840331/ [5840333] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps} time: 00:23:43 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840333/ [5840334] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/repair_test} time: 00:14:08 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840334/ [5840335] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:08:13 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840335/ [5840336] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_cephadm_repos} time: 00:06:35 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840336/ [5840337] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:12:01 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840337/ [5840338] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/mon_recovery validater/lockdep} time: 00:12:46 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840338/ [5840340] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:09:42 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840340/ [5840341] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/octopus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/none thrashosds-health workloads/cache-snaps} time: 00:25:45 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840341/ [5840342] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:47 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840342/ [5840343] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-snaps} time: 00:18:23 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840343/ [5840344] rados/singleton/{all/erasure-code-nonregression mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} time: 00:08:43 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840344/ [5840345] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/connectivity start} time: 00:13:26 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840345/ [5840346] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/cache} time: 00:12:44 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840346/ [5840347] rados/multimon/{clusters/9 mon_election/classic msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:05:27 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840347/ [5840348] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:10:23 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840348/ [5840349] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/pool-snaps-few-objects} time: 00:26:35 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840349/ [5840350] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:21:45 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840350/ [5840351] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:11:16 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840351/ [5840352] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/rados_api_tests} time: 00:29:27 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840352/ [5840353] rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/one workloads/rados_api_tests} time: 00:34:59 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840353/ [5840354] rados/singleton/{all/lost-unfound-delete mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8}} time: 00:53:47 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840354/ [5840355] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:28:02 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840355/ [5840356] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/pacific backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/pggrow thrashosds-health workloads/radosbench} time: 01:35:12 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840356/ [5840357] rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/misc} time: 00:30:15 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840357/ [5840358] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} tasks/failover} time: 00:10:54 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840358/ [5840360] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} tasks/dashboard} time: 01:10:25 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840360/ [5840361] rados/objectstore/{backends/filejournal supported-random-distro$/{centos_8}} time: 00:09:23 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840361/ [5840362] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_orch_cli} time: 00:19:37 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840362/ [5840363] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/radosbench} time: 02:30:46 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840363/ [5840364] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:31:47 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840364/ [5840365] rados/cephadm/smoke/{distro/ubuntu_20.04 fixed-2 mon_election/classic start} time: 00:14:26 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840365/ [5840366] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:06:13 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840366/ [5840367] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/redirect} time: 00:12:08 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840367/ [5840368] rados/singleton/{all/lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}} time: 00:48:51 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840368/ [5840369] rados/cephadm/smoke-roleless/{distro/rhel_latest start} time: 00:12:12 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840369/ [5840370] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/redirect_promote_tests} time: 00:17:13 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840370/ [5840371] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} tasks/scrub_test} time: 00:12:19 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840371/ [5840372] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:32:08 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840372/ [5840373] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/rbd_cls} time: 00:20:21 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840373/ [5840374] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:22:21 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840374/ [5840375] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/redirect_set_object} time: 00:17:40 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840375/ [5840376] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:09:03 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840376/ [5840377] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:07:02 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840377/ [5840378] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/set-chunks-read} time: 00:18:33 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840378/ [5840380] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:09:06 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840380/ [5840381] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:28:31 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840381/ [5840382] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados tasks/rados_api_tests validater/valgrind} time: 00:47:14 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840382/ [5840383] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:12 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840383/ [5840384] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/small-objects-balanced} time: 00:20:27 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840384/ [5840385] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/connectivity start} time: 00:14:29 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840385/ [5840386] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:09:45 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840386/ [5840387] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/small-objects-localized} time: 00:21:01 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840387/ [5840389] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{centos_latest} fixed-2 mon_election/classic} time: 00:18:38 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840389/ [5840390] rados/multimon/{clusters/21 mon_election/connectivity msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} tasks/mon_recovery} time: 00:22:01 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840390/ [5840391] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/small-objects} time: 00:21:03 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840391/ [5840392] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:14:48 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840392/ [5840393] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:10:08 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840393/ [5840394] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_adoption} time: 00:09:12 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840394/ [5840395] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-balanced} time: 00:39:30 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840395/ [5840396] rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/sync-many workloads/rados_mon_osdmap_prune} time: 00:42:59 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840396/ [5840397] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:27:49 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840397/ [5840398] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/careful thrashosds-health workloads/snaps-few-objects} time: 00:38:11 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840398/ [5840399] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/snaps-few-objects-localized} time: 00:29:16 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840399/ [5840400] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/filestore-xfs supported-random-distro$/{rhel_8} tasks/insights} time: 00:14:08 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840400/ [5840401] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/test_rbd_api} time: 00:19:31 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840401/ [5840402] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/snaps-few-objects} time: 00:30:27 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840402/ [5840403] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/classic msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8}} time: 00:13:14 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840403/ [5840404] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:09:55 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840404/ [5840405] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:14:54 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840405/ [5840406] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:23:40 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840406/ [5840407] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/classic start} time: 00:13:58 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840407/ [5840408] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/write_fadvise_dontneed} time: 00:17:47 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840408/ [5840409] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} tasks/rados_api_tests} time: 00:27:57 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840409/ [5840410] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} tasks/dashboard} time: 01:10:15 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840410/ [5840411] rados/objectstore/{backends/filestore-idempotent-aio-journal supported-random-distro$/{ubuntu_latest}} time: 02:28:26 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840411/ [5840412] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} time: 00:20:31 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840412/ [5840413] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/admin_socket_objecter_requests} time: 00:41:01 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840413/ [5840414] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04 start} time: 00:11:34 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840414/ [5840415] rados/singleton/{all/mon-auth-caps mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 00:10:06 info: http://pulpito.ceph.com/kchai-2021-01-29_12:57:23-rados-wip-kefu-testing-2021-01-29-1713-distro-basic-smithi/5840415/