Found 367 jobs Ignoring malparsed backtrace: ceph version 16.0.0-8592-g3ef56c39 (3ef56c398ba628e9011753b67786b8e2a37447a6) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f1f1b125a2b] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x56176cc39d60] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-12-28T12:08:49.684 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-12-28T12:08:50.022 INFO:teuthology.orchestra.run.smithi179.stdout: Ignoring malparsed backtrace: 2020-12-28T12:10:12.020 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-28T12:10:12.103 INFO:tasks.ceph.osd.4:Stopped , 2020-12-28T12:10:12.103 DEBUG:tasks.ceph.osd.5:waiting for process to exit Ignoring malparsed backtrace: 2020-12-28T12:12:01.785 INFO:teuthology.orchestra.run.smithi179.stdout: remote refid st t when poll reach delay offset jitter , 2020-12-28T12:12:01.786 INFO:teuthology.orchestra.run.smithi179.stdout:============================================================================== , 2020-12-28T12:12:01.786 INFO:teuthology.orchestra.run.smithi179.stdout:+hv01.front.sepi 216.240.36.24 3 u 7 64 377 0.069 -7.145 7.272 Ignoring malparsed backtrace: ceph version 16.0.0-8592-g3ef56c39 (3ef56c398ba628e9011753b67786b8e2a37447a6) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f1f1b125a2b] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x56176cc39d60] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-12-28T12:08:49.684 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-12-28T12:08:50.022 INFO:teuthology.orchestra.run.smithi179.stdout: Ignoring malparsed backtrace: 2020-12-28T12:10:12.020 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-28T12:10:12.103 INFO:tasks.ceph.osd.4:Stopped , 2020-12-28T12:10:12.103 DEBUG:tasks.ceph.osd.5:waiting for process to exit Ignoring malparsed backtrace: 2020-12-28T12:12:01.785 INFO:teuthology.orchestra.run.smithi179.stdout: remote refid st t when poll reach delay offset jitter , 2020-12-28T12:12:01.786 INFO:teuthology.orchestra.run.smithi179.stdout:============================================================================== , 2020-12-28T12:12:01.786 INFO:teuthology.orchestra.run.smithi179.stdout:+hv01.front.sepi 216.240.36.24 3 u 7 64 377 0.069 -7.145 7.272 Ignoring malparsed backtrace: ceph version 16.0.0-8592-g3ef56c39 (3ef56c398ba628e9011753b67786b8e2a37447a6) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f1f1b125a2b] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x56176cc39d60] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-12-28T12:08:49.684 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-12-28T12:08:50.022 INFO:teuthology.orchestra.run.smithi179.stdout: Ignoring malparsed backtrace: 2020-12-28T12:10:12.020 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-28T12:10:12.103 INFO:tasks.ceph.osd.4:Stopped , 2020-12-28T12:10:12.103 DEBUG:tasks.ceph.osd.5:waiting for process to exit Ignoring malparsed backtrace: 2020-12-28T12:12:01.785 INFO:teuthology.orchestra.run.smithi179.stdout: remote refid st t when poll reach delay offset jitter , 2020-12-28T12:12:01.786 INFO:teuthology.orchestra.run.smithi179.stdout:============================================================================== , 2020-12-28T12:12:01.786 INFO:teuthology.orchestra.run.smithi179.stdout:+hv01.front.sepi 216.240.36.24 3 u 7 64 377 0.069 -7.145 7.272 Ignoring malparsed backtrace: ceph version 16.0.0-8592-g3ef56c39 (3ef56c398ba628e9011753b67786b8e2a37447a6) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f8c99ebda2b] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x59c) [0x565213056c4c] Ignoring malparsed backtrace: 1 , 2 , abort Ignoring malparsed backtrace: 2020-12-28T10:29:39.158 INFO:teuthology.orchestra.run.smithi092.stdout:64424509477 , 2020-12-28T10:29:39.173 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509477 got 64424509477 for osd.1 , 2020-12-28T10:29:39.173 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 Ignoring malparsed backtrace: dumped all , 2020-12-28T10:30:03.114 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2020-12-28T10:30:02.014517+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22591064938,"num_objects":7584,"num_object_clones":0,"num_object_copies":26392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5597,"num_whiteouts":3,"num_read":11198,"num_read_kb":5604,"num_write":30588,"num_write_kb":18042095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":830,"num_bytes_recovered":2318619681,"num_keys_recovered":2,"num_objects_omap":3,"num_objects_hit_set_archive":21,"num_bytes_hit_set_archive":12303,"num_flush":3643,"num_flush_kb":10623124,"num_evict":1682,"num_evict_kb":4889582,"num_promote":5602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15669,"ondisk_log_size":15669,"up":115,"acting":113,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":115,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":31045320,"kb_used_data":30614768,"kb_used_omap":0,"kb_used_meta":419583,"kb_avail":723929400,"statfs":{"total":773094113280,"available":741303705600,"internally_reserved":0,"allocated":31349522432,"data_stored":37113846502,"data_compressed":602711681,"data_compressed_allocated":5885046784,"data_compressed_original":11770093568,"omap_allocated":964,"internal_metadata":429653052},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-355,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-363,"num_bytes_recovered":-1229979446,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002045"},"pg_stats":[{"pgid":"2.11","version":"0'0","reported_seq":"76","reported_epoch":"117","state":"active+clean","last_fresh":"2020-12-28T10:29:57.117116+0000","last_change":"2020-12-28T10:29:18.165947+0000","last_active":"2020-12-28T10:29:57.117116+0000","last_peered":"2020-12-28T10:29:57.117116+0000","last_clean":"2020-12-28T10:29:57.117116+0000","last_became_active":"2020-12-28T10:29:18.165439+0000","last_became_peered":"2020-12-28T10:29:18.165439+0000","last_unstale":"2020-12-28T10:29:57.117116+0000","last_undegraded":"2020-12-28T10:29:57.117116+0000","last_fullsized":"2020-12-28T10:29:57.117116+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:28:00.453819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:28:00.453819+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"92'7542","reported_seq":"10152","reported_epoch":"118","state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2020-12-28T10:29:58.481266+0000","last_change":"2020-12-28T10:29:56.272678+0000","last_active":"2020-12-28T10:29:58.481266+0000","last_peered":"2020-12-28T10:29:58.481266+0000","last_clean":"2020-12-28T10:28:10.052115+0000","last_became_active":"2020-12-28T10:29:19.388633+0000","last_became_peered":"2020-12-28T10:29:19.388633+0000","last_unstale":"2020-12-28T10:29:58.481266+0000","last_undegraded":"2020-12-28T10:29:18.926010+0000","last_fullsized":"2020-12-28T10:29:18.911586+0000","mapping_epoch":95,"log_start":"75'4541","ondisk_log_start":"75'4541","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:27:10.167150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:27:10.167150+0000","last_clean_scrub_stamp":"2020-12-28T10:27:10.167150+0000","log_size":3001,"ondisk_log_size":3001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2955629649,"num_objects":984,"num_object_clones":0,"num_object_copies":2952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":886,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":489,"num_whiteouts":0,"num_read":2842,"num_read_kb":1421,"num_write":6866,"num_write_kb":1887177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":93724420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1946,"num_flush":933,"num_flush_kb":2735053,"num_evict":443,"num_evict_kb":1287494,"num_promote":1423,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":984}],"blocked_by":[],"up_primary":4,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"120","reported_epoch":"117","state":"active+clean","last_fresh":"2020-12-28T10:29:57.118302+0000","last_change":"2020-12-28T10:28:25.647913+0000","last_active":"2020-12-28T10:29:57.118302+0000","last_peered":"2020-12-28T10:29:57.118302+0000","last_clean":"2020-12-28T10:29:57.118302+0000","last_became_active":"2020-12-28T10:28:07.912797+0000","last_became_peered":"2020-12-28T10:28:07.912797+0000","last_unstale":"2020-12-28T10:29:57.118302+0000","last_undegraded":"2020-12-28T10:29:57.118302+0000","last_fullsized":"2020-12-28T10:29:57.118302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:28:25.647771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:28:25.647771+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"17'2","reported_seq":"85","reported_epoch":"117","state":"active+clean","last_fresh":"2020-12-28T10:29:57.099548+0000","last_change":"2020-12-28T10:28:12.706409+0000","last_active":"2020-12-28T10:29:57.099548+0000","last_peered":"2020-12-28T10:29:57.099548+0000","last_clean":"2020-12-28T10:29:57.099548+0000","last_became_active":"2020-12-28T10:28:06.931370+0000","last_became_peered":"2020-12-28T10:28:06.931370+0000","last_unstale":"2020-12-28T10:29:57.099548+0000","last_undegraded":"2020-12-28T10:29:57.099548+0000","last_fullsized":"2020-12-28T10:29:57.099548+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":3,"last_scrub":"17'2","last_scrub_stamp":"2020-12-28T10:28:12.706274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:44.957484+0000","last_clean_scrub_stamp":"2020-12-28T10:28:12.706274+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"90'7269","reported_seq":"9692","reported_epoch":"117","state":"active+clean","last_fresh":"2020-12-28T10:29:57.099127+0000","last_change":"2020-12-28T10:29:05.014780+0000","last_active":"2020-12-28T10:29:57.099127+0000","last_peered":"2020-12-28T10:29:57.099127+0000","last_clean":"2020-12-28T10:29:57.099127+0000","last_became_active":"2020-12-28T10:27:11.629397+0000","last_became_peered":"2020-12-28T10:27:11.629397+0000","last_unstale":"2020-12-28T10:29:57.099127+0000","last_undegraded":"2020-12-28T10:29:57.099127+0000","last_fullsized":"2020-12-28T10:29:57.099127+0000","mapping_epoch":26,"log_start":"74'4268","ondisk_log_start":"74'4268","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'7202","last_scrub_stamp":"2020-12-28T10:29:05.014618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:27:10.167150+0000","last_clean_scrub_stamp":"2020-12-28T10:29:05.014618+0000","log_size":3001,"ondisk_log_size":3001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2905734603,"num_objects":986,"num_object_clones":0,"num_object_copies":2958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":2803,"num_read_kb":1402,"num_write":6741,"num_write_kb":1851740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":3558,"num_flush":913,"num_flush_kb":2644122,"num_evict":421,"num_evict_kb":1233889,"num_promote":1401,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"130","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:54.207517+0000","last_change":"2020-12-28T10:29:18.168739+0000","last_active":"2020-12-28T10:29:54.207517+0000","last_peered":"2020-12-28T10:29:54.207517+0000","last_clean":"2020-12-28T10:29:54.207517+0000","last_became_active":"2020-12-28T10:29:18.168127+0000","last_became_peered":"2020-12-28T10:29:18.168127+0000","last_unstale":"2020-12-28T10:29:54.207517+0000","last_undegraded":"2020-12-28T10:29:54.207517+0000","last_fullsized":"2020-12-28T10:29:54.207517+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:26:53.017342+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"17'2","reported_seq":"52","reported_epoch":"118","state":"active+clean","last_fresh":"2020-12-28T10:29:58.110371+0000","last_change":"2020-12-28T10:29:31.932411+0000","last_active":"2020-12-28T10:29:58.110371+0000","last_peered":"2020-12-28T10:29:58.110371+0000","last_clean":"2020-12-28T10:29:58.110371+0000","last_became_active":"2020-12-28T10:29:18.168215+0000","last_became_peered":"2020-12-28T10:29:18.168215+0000","last_unstale":"2020-12-28T10:29:58.110371+0000","last_undegraded":"2020-12-28T10:29:58.110371+0000","last_fullsized":"2020-12-28T10:29:58.110371+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:26:44.957484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:44.957484+0000","last_clean_scrub_stamp":"2020-12-28T10:26:44.957484+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"131","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:54.211703+0000","last_change":"2020-12-28T10:29:18.175803+0000","last_active":"2020-12-28T10:29:54.211703+0000","last_peered":"2020-12-28T10:29:54.211703+0000","last_clean":"2020-12-28T10:29:54.211703+0000","last_became_active":"2020-12-28T10:29:18.174884+0000","last_became_peered":"2020-12-28T10:29:18.174884+0000","last_unstale":"2020-12-28T10:29:54.211703+0000","last_undegraded":"2020-12-28T10:29:54.211703+0000","last_fullsized":"2020-12-28T10:29:54.211703+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:26:53.017342+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"90'7052","reported_seq":"9397","reported_epoch":"117","state":"active+clean","last_fresh":"2020-12-28T10:29:57.129776+0000","last_change":"2020-12-28T10:28:43.058209+0000","last_active":"2020-12-28T10:29:57.129776+0000","last_peered":"2020-12-28T10:29:57.129776+0000","last_clean":"2020-12-28T10:29:57.129776+0000","last_became_active":"2020-12-28T10:27:11.629316+0000","last_became_peered":"2020-12-28T10:27:11.629316+0000","last_unstale":"2020-12-28T10:29:57.129776+0000","last_undegraded":"2020-12-28T10:29:57.129776+0000","last_fullsized":"2020-12-28T10:29:57.129776+0000","mapping_epoch":26,"log_start":"72'4051","ondisk_log_start":"72'4051","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"81'5567","last_scrub_stamp":"2020-12-28T10:28:43.058063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:27:10.167150+0000","last_clean_scrub_stamp":"2020-12-28T10:28:43.058063+0000","log_size":3001,"ondisk_log_size":3001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2935489361,"num_objects":986,"num_object_clones":0,"num_object_copies":2958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":1,"num_read":2722,"num_read_kb":1362,"num_write":6536,"num_write_kb":1803479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":3472,"num_flush":872,"num_flush_kb":2554180,"num_evict":382,"num_evict_kb":1123715,"num_promote":1362,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"17'2","reported_seq":"91","reported_epoch":"117","state":"active+clean","last_fresh":"2020-12-28T10:29:57.118154+0000","last_change":"2020-12-28T10:28:13.720167+0000","last_active":"2020-12-28T10:29:57.118154+0000","last_peered":"2020-12-28T10:29:57.118154+0000","last_clean":"2020-12-28T10:29:57.118154+0000","last_became_active":"2020-12-28T10:28:06.926449+0000","last_became_peered":"2020-12-28T10:28:06.926449+0000","last_unstale":"2020-12-28T10:29:57.118154+0000","last_undegraded":"2020-12-28T10:29:57.118154+0000","last_fullsized":"2020-12-28T10:29:57.118154+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":3,"last_scrub":"17'2","last_scrub_stamp":"2020-12-28T10:28:13.720041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:44.957484+0000","last_clean_scrub_stamp":"2020-12-28T10:28:13.720041+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"92'933","reported_seq":"1022","reported_epoch":"117","state":"active+clean","last_fresh":"2020-12-28T10:29:57.099735+0000","last_change":"2020-12-28T10:28:54.178003+0000","last_active":"2020-12-28T10:29:57.099735+0000","last_peered":"2020-12-28T10:29:57.099735+0000","last_clean":"2020-12-28T10:29:57.099735+0000","last_became_active":"2020-12-28T10:27:40.224828+0000","last_became_peered":"2020-12-28T10:27:40.224828+0000","last_unstale":"2020-12-28T10:29:57.099735+0000","last_undegraded":"2020-12-28T10:29:57.099735+0000","last_fullsized":"2020-12-28T10:29:57.099735+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'626","last_scrub_stamp":"2020-12-28T10:28:54.177869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:27:07.133282+0000","last_clean_scrub_stamp":"2020-12-28T10:28:54.177869+0000","log_size":933,"ondisk_log_size":933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2796541282,"num_objects":932,"num_object_clones":0,"num_object_copies":3728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":933,"num_write_kb":2735053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0,2],"acting":[3,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"120","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:54.211175+0000","last_change":"2020-12-28T10:28:17.558424+0000","last_active":"2020-12-28T10:29:54.211175+0000","last_peered":"2020-12-28T10:29:54.211175+0000","last_clean":"2020-12-28T10:29:54.211175+0000","last_became_active":"2020-12-28T10:28:07.941979+0000","last_became_peered":"2020-12-28T10:28:07.941979+0000","last_unstale":"2020-12-28T10:29:54.211175+0000","last_undegraded":"2020-12-28T10:29:54.211175+0000","last_fullsized":"2020-12-28T10:29:54.211175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:28:17.558340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:28:17.558340+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"17'2","reported_seq":"89","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:54.207524+0000","last_change":"2020-12-28T10:29:18.170020+0000","last_active":"2020-12-28T10:29:54.207524+0000","last_peered":"2020-12-28T10:29:54.207524+0000","last_clean":"2020-12-28T10:29:54.207524+0000","last_became_active":"2020-12-28T10:29:18.168271+0000","last_became_peered":"2020-12-28T10:29:18.168271+0000","last_unstale":"2020-12-28T10:29:54.207524+0000","last_undegraded":"2020-12-28T10:29:54.207524+0000","last_fullsized":"2020-12-28T10:29:54.207524+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:26:44.957484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:44.957484+0000","last_clean_scrub_stamp":"2020-12-28T10:26:44.957484+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":1,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"91'925","reported_seq":"2541","reported_epoch":"118","state":"peering","last_fresh":"2020-12-28T10:29:58.106604+0000","last_change":"2020-12-28T10:29:58.107212+0000","last_active":"2020-12-28T10:29:58.106604+0000","last_peered":"2020-12-28T10:29:57.457643+0000","last_clean":"2020-12-28T10:28:10.039532+0000","last_became_active":"2020-12-28T10:29:19.399774+0000","last_became_peered":"2020-12-28T10:29:19.399774+0000","last_unstale":"2020-12-28T10:29:58.106604+0000","last_undegraded":"2020-12-28T10:29:58.106604+0000","last_fullsized":"2020-12-28T10:29:58.106604+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:27:07.133282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:27:07.133282+0000","last_clean_scrub_stamp":"2020-12-28T10:27:07.133282+0000","log_size":925,"ondisk_log_size":925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2749201154,"num_objects":923,"num_object_clones":0,"num_object_copies":3692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":925,"num_write_kb":2689769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":742,"num_bytes_recovered":2201625494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6,4],"acting":[3,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5,6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'2","reported_seq":"116","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:55.535631+0000","last_change":"2020-12-28T10:29:55.535631+0000","last_active":"2020-12-28T10:29:55.535631+0000","last_peered":"2020-12-28T10:29:55.535631+0000","last_clean":"2020-12-28T10:29:55.535631+0000","last_became_active":"2020-12-28T10:28:02.003741+0000","last_became_peered":"2020-12-28T10:28:02.003741+0000","last_unstale":"2020-12-28T10:29:55.535631+0000","last_undegraded":"2020-12-28T10:29:55.535631+0000","last_fullsized":"2020-12-28T10:29:55.535631+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'2","last_scrub_stamp":"2020-12-28T10:28:02.525871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:44.957484+0000","last_clean_scrub_stamp":"2020-12-28T10:28:02.525871+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"121","reported_epoch":"117","state":"active+clean","last_fresh":"2020-12-28T10:29:57.130515+0000","last_change":"2020-12-28T10:28:38.757039+0000","last_active":"2020-12-28T10:29:57.130515+0000","last_peered":"2020-12-28T10:29:57.130515+0000","last_clean":"2020-12-28T10:29:57.130515+0000","last_became_active":"2020-12-28T10:28:07.929443+0000","last_became_peered":"2020-12-28T10:28:07.929443+0000","last_unstale":"2020-12-28T10:29:57.130515+0000","last_undegraded":"2020-12-28T10:29:57.130515+0000","last_fullsized":"2020-12-28T10:29:57.130515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:28:38.756489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:28:38.756489+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"90'913","reported_seq":"1033","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:54.207304+0000","last_change":"2020-12-28T10:28:55.869392+0000","last_active":"2020-12-28T10:29:54.207304+0000","last_peered":"2020-12-28T10:29:54.207304+0000","last_clean":"2020-12-28T10:29:54.207304+0000","last_became_active":"2020-12-28T10:27:10.283559+0000","last_became_peered":"2020-12-28T10:27:10.283559+0000","last_unstale":"2020-12-28T10:29:54.207304+0000","last_undegraded":"2020-12-28T10:29:54.207304+0000","last_fullsized":"2020-12-28T10:29:54.207304+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'735","last_scrub_stamp":"2020-12-28T10:28:55.869260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:27:07.133282+0000","last_clean_scrub_stamp":"2020-12-28T10:28:55.869260+0000","log_size":913,"ondisk_log_size":913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2707114918,"num_objects":913,"num_object_clones":0,"num_object_copies":3652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":913,"num_write_kb":2644122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,5],"acting":[1,2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"66","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:53.911751+0000","last_change":"2020-12-28T10:29:26.273156+0000","last_active":"2020-12-28T10:29:53.911751+0000","last_peered":"2020-12-28T10:29:53.911751+0000","last_clean":"2020-12-28T10:29:53.911751+0000","last_became_active":"2020-12-28T10:29:26.272353+0000","last_became_peered":"2020-12-28T10:29:26.272353+0000","last_unstale":"2020-12-28T10:29:53.911751+0000","last_undegraded":"2020-12-28T10:29:53.911751+0000","last_fullsized":"2020-12-28T10:29:53.911751+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:28:49.962084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:28:49.962084+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'2","reported_seq":"119","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:54.210812+0000","last_change":"2020-12-28T10:29:18.175552+0000","last_active":"2020-12-28T10:29:54.210812+0000","last_peered":"2020-12-28T10:29:54.210812+0000","last_clean":"2020-12-28T10:29:54.210812+0000","last_became_active":"2020-12-28T10:29:18.174850+0000","last_became_peered":"2020-12-28T10:29:18.174850+0000","last_unstale":"2020-12-28T10:29:54.210812+0000","last_undegraded":"2020-12-28T10:29:54.210812+0000","last_fullsized":"2020-12-28T10:29:54.210812+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:26:44.957484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:44.957484+0000","last_clean_scrub_stamp":"2020-12-28T10:26:44.957484+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'872","reported_seq":"985","reported_epoch":"118","state":"active+clean","last_fresh":"2020-12-28T10:29:58.124032+0000","last_change":"2020-12-28T10:29:58.122528+0000","last_active":"2020-12-28T10:29:58.124032+0000","last_peered":"2020-12-28T10:29:58.124032+0000","last_clean":"2020-12-28T10:29:58.124032+0000","last_became_active":"2020-12-28T10:29:58.121167+0000","last_became_peered":"2020-12-28T10:29:58.121167+0000","last_unstale":"2020-12-28T10:29:58.124032+0000","last_undegraded":"2020-12-28T10:29:58.124032+0000","last_fullsized":"2020-12-28T10:29:58.124032+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:27:07.133282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:27:07.133282+0000","last_clean_scrub_stamp":"2020-12-28T10:27:07.133282+0000","log_size":872,"ondisk_log_size":872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2615028768,"num_objects":872,"num_object_clones":0,"num_object_copies":3488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":872,"num_write_kb":2554180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,6],"acting":[4,5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"125","reported_epoch":"117","state":"active+clean","last_fresh":"2020-12-28T10:29:57.131705+0000","last_change":"2020-12-28T10:28:49.690414+0000","last_active":"2020-12-28T10:29:57.131705+0000","last_peered":"2020-12-28T10:29:57.131705+0000","last_clean":"2020-12-28T10:29:57.131705+0000","last_became_active":"2020-12-28T10:28:07.936804+0000","last_became_peered":"2020-12-28T10:28:07.936804+0000","last_unstale":"2020-12-28T10:29:57.131705+0000","last_undegraded":"2020-12-28T10:29:57.131705+0000","last_fullsized":"2020-12-28T10:29:57.131705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:28:49.690271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:28:49.690271+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"17'2","reported_seq":"90","reported_epoch":"117","state":"active+clean","last_fresh":"2020-12-28T10:29:57.130391+0000","last_change":"2020-12-28T10:28:37.712031+0000","last_active":"2020-12-28T10:29:57.130391+0000","last_peered":"2020-12-28T10:29:57.130391+0000","last_clean":"2020-12-28T10:29:57.130391+0000","last_became_active":"2020-12-28T10:28:06.970929+0000","last_became_peered":"2020-12-28T10:28:06.970929+0000","last_unstale":"2020-12-28T10:29:57.130391+0000","last_undegraded":"2020-12-28T10:29:57.130391+0000","last_fullsized":"2020-12-28T10:29:57.130391+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":4,"last_scrub":"17'2","last_scrub_stamp":"2020-12-28T10:28:37.711820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:44.957484+0000","last_clean_scrub_stamp":"2020-12-28T10:28:37.711820+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"91'7447","reported_seq":"9940","reported_epoch":"117","state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2020-12-28T10:29:58.082741+0000","last_change":"2020-12-28T10:29:57.458678+0000","last_active":"2020-12-28T10:29:58.082741+0000","last_peered":"2020-12-28T10:29:58.082741+0000","last_clean":"2020-12-28T10:28:10.150690+0000","last_became_active":"2020-12-28T10:29:19.419249+0000","last_became_peered":"2020-12-28T10:29:19.419249+0000","last_unstale":"2020-12-28T10:29:58.082741+0000","last_undegraded":"2020-12-28T10:29:18.914771+0000","last_fullsized":"2020-12-28T10:29:18.909895+0000","mapping_epoch":95,"log_start":"75'4446","ondisk_log_start":"75'4446","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:27:10.167150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:27:10.167150+0000","last_clean_scrub_stamp":"2020-12-28T10:27:10.167150+0000","log_size":3001,"ondisk_log_size":3001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2926325203,"num_objects":985,"num_object_clones":0,"num_object_copies":2955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":489,"num_whiteouts":2,"num_read":2831,"num_read_kb":1419,"num_write":6801,"num_write_kb":1876574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":23269767,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":3327,"num_flush":925,"num_flush_kb":2689769,"num_evict":436,"num_evict_kb":1244484,"num_promote":1416,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":985}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"123","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:54.207482+0000","last_change":"2020-12-28T10:28:12.145089+0000","last_active":"2020-12-28T10:29:54.207482+0000","last_peered":"2020-12-28T10:29:54.207482+0000","last_clean":"2020-12-28T10:29:54.207482+0000","last_became_active":"2020-12-28T10:28:07.941514+0000","last_became_peered":"2020-12-28T10:28:07.941514+0000","last_unstale":"2020-12-28T10:29:54.207482+0000","last_undegraded":"2020-12-28T10:29:54.207482+0000","last_fullsized":"2020-12-28T10:29:54.207482+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:28:12.144923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:28:12.144923+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"17'2","reported_seq":"91","reported_epoch":"117","state":"active+clean","last_fresh":"2020-12-28T10:29:57.118422+0000","last_change":"2020-12-28T10:28:12.741372+0000","last_active":"2020-12-28T10:29:57.118422+0000","last_peered":"2020-12-28T10:29:57.118422+0000","last_clean":"2020-12-28T10:29:57.118422+0000","last_became_active":"2020-12-28T10:28:06.934451+0000","last_became_peered":"2020-12-28T10:28:06.934451+0000","last_unstale":"2020-12-28T10:29:57.118422+0000","last_undegraded":"2020-12-28T10:29:57.118422+0000","last_fullsized":"2020-12-28T10:29:57.118422+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":3,"last_scrub":"17'2","last_scrub_stamp":"2020-12-28T10:28:12.741288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:44.957484+0000","last_clean_scrub_stamp":"2020-12-28T10:28:12.741288+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"17'2","reported_seq":"73","reported_epoch":"117","state":"active+clean","last_fresh":"2020-12-28T10:29:57.130175+0000","last_change":"2020-12-28T10:28:16.738909+0000","last_active":"2020-12-28T10:29:57.130175+0000","last_peered":"2020-12-28T10:29:57.130175+0000","last_clean":"2020-12-28T10:29:57.130175+0000","last_became_active":"2020-12-28T10:28:06.933486+0000","last_became_peered":"2020-12-28T10:28:06.933486+0000","last_unstale":"2020-12-28T10:29:57.130175+0000","last_undegraded":"2020-12-28T10:29:57.130175+0000","last_fullsized":"2020-12-28T10:29:57.130175+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":4,"last_scrub":"17'2","last_scrub_stamp":"2020-12-28T10:28:16.738750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:44.957484+0000","last_clean_scrub_stamp":"2020-12-28T10:28:16.738750+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"131","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:54.211507+0000","last_change":"2020-12-28T10:29:18.179872+0000","last_active":"2020-12-28T10:29:54.211507+0000","last_peered":"2020-12-28T10:29:54.211507+0000","last_clean":"2020-12-28T10:29:54.211507+0000","last_became_active":"2020-12-28T10:29:18.179291+0000","last_became_peered":"2020-12-28T10:29:18.179291+0000","last_unstale":"2020-12-28T10:29:54.211507+0000","last_undegraded":"2020-12-28T10:29:54.211507+0000","last_fullsized":"2020-12-28T10:29:54.211507+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:26:53.017342+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.9","version":"17'2","reported_seq":"113","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:54.210824+0000","last_change":"2020-12-28T10:28:14.506379+0000","last_active":"2020-12-28T10:29:54.210824+0000","last_peered":"2020-12-28T10:29:54.210824+0000","last_clean":"2020-12-28T10:29:54.210824+0000","last_became_active":"2020-12-28T10:28:06.929521+0000","last_became_peered":"2020-12-28T10:28:06.929521+0000","last_unstale":"2020-12-28T10:29:54.210824+0000","last_undegraded":"2020-12-28T10:29:54.210824+0000","last_fullsized":"2020-12-28T10:29:54.210824+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":4,"last_scrub":"17'2","last_scrub_stamp":"2020-12-28T10:28:14.506230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:44.957484+0000","last_clean_scrub_stamp":"2020-12-28T10:28:14.506230+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"127","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:54.206797+0000","last_change":"2020-12-28T10:28:38.540211+0000","last_active":"2020-12-28T10:29:54.206797+0000","last_peered":"2020-12-28T10:29:54.206797+0000","last_clean":"2020-12-28T10:29:54.206797+0000","last_became_active":"2020-12-28T10:28:09.918343+0000","last_became_peered":"2020-12-28T10:28:09.918343+0000","last_unstale":"2020-12-28T10:29:54.206797+0000","last_undegraded":"2020-12-28T10:29:54.206797+0000","last_fullsized":"2020-12-28T10:29:54.206797+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:28:38.540092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:28:38.540092+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"17'2","reported_seq":"73","reported_epoch":"117","state":"active+clean","last_fresh":"2020-12-28T10:29:57.131381+0000","last_change":"2020-12-28T10:28:17.736899+0000","last_active":"2020-12-28T10:29:57.131381+0000","last_peered":"2020-12-28T10:29:57.131381+0000","last_clean":"2020-12-28T10:29:57.131381+0000","last_became_active":"2020-12-28T10:28:06.945387+0000","last_became_peered":"2020-12-28T10:28:06.945387+0000","last_unstale":"2020-12-28T10:29:57.131381+0000","last_undegraded":"2020-12-28T10:29:57.131381+0000","last_fullsized":"2020-12-28T10:29:57.131381+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":4,"last_scrub":"17'2","last_scrub_stamp":"2020-12-28T10:28:17.736735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:44.957484+0000","last_clean_scrub_stamp":"2020-12-28T10:28:17.736735+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"64","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:54.206791+0000","last_change":"2020-12-28T10:28:16.081882+0000","last_active":"2020-12-28T10:29:54.206791+0000","last_peered":"2020-12-28T10:29:54.206791+0000","last_clean":"2020-12-28T10:29:54.206791+0000","last_became_active":"2020-12-28T10:28:09.927743+0000","last_became_peered":"2020-12-28T10:28:09.927743+0000","last_unstale":"2020-12-28T10:29:54.206791+0000","last_undegraded":"2020-12-28T10:29:54.206791+0000","last_fullsized":"2020-12-28T10:29:54.206791+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:28:16.081778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:28:16.081778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"128","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:54.210958+0000","last_change":"2020-12-28T10:28:39.659681+0000","last_active":"2020-12-28T10:29:54.210958+0000","last_peered":"2020-12-28T10:29:54.210958+0000","last_clean":"2020-12-28T10:29:54.210958+0000","last_became_active":"2020-12-28T10:28:09.929292+0000","last_became_peered":"2020-12-28T10:28:09.929292+0000","last_unstale":"2020-12-28T10:29:54.210958+0000","last_undegraded":"2020-12-28T10:29:54.210958+0000","last_fullsized":"2020-12-28T10:29:54.210958+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:28:39.659522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:28:39.659522+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":"27","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:53.911569+0000","last_change":"2020-12-28T10:29:26.272880+0000","last_active":"2020-12-28T10:29:53.911569+0000","last_peered":"2020-12-28T10:29:53.911569+0000","last_clean":"2020-12-28T10:29:53.911569+0000","last_became_active":"2020-12-28T10:29:26.272244+0000","last_became_peered":"2020-12-28T10:29:26.272244+0000","last_unstale":"2020-12-28T10:29:53.911569+0000","last_undegraded":"2020-12-28T10:29:53.911569+0000","last_fullsized":"2020-12-28T10:29:53.911569+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:28:50.631577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:28:50.631577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"75","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:54.207512+0000","last_change":"2020-12-28T10:29:18.168818+0000","last_active":"2020-12-28T10:29:54.207512+0000","last_peered":"2020-12-28T10:29:54.207512+0000","last_clean":"2020-12-28T10:29:54.207512+0000","last_became_active":"2020-12-28T10:29:18.168241+0000","last_became_peered":"2020-12-28T10:29:18.168241+0000","last_unstale":"2020-12-28T10:29:54.207512+0000","last_undegraded":"2020-12-28T10:29:54.207512+0000","last_fullsized":"2020-12-28T10:29:54.207512+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:26:53.017342+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":"53","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:56.139746+0000","last_change":"2020-12-28T10:29:56.139746+0000","last_active":"2020-12-28T10:29:56.139746+0000","last_peered":"2020-12-28T10:29:56.139746+0000","last_clean":"2020-12-28T10:29:56.139746+0000","last_became_active":"2020-12-28T10:29:26.604318+0000","last_became_peered":"2020-12-28T10:29:26.604318+0000","last_unstale":"2020-12-28T10:29:56.139746+0000","last_undegraded":"2020-12-28T10:29:56.139746+0000","last_fullsized":"2020-12-28T10:29:56.139746+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:28:50.965787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:28:50.965787+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":"47","reported_epoch":"118","state":"active+clean","last_fresh":"2020-12-28T10:29:58.110706+0000","last_change":"2020-12-28T10:29:31.937831+0000","last_active":"2020-12-28T10:29:58.110706+0000","last_peered":"2020-12-28T10:29:58.110706+0000","last_clean":"2020-12-28T10:29:58.110706+0000","last_became_active":"2020-12-28T10:29:18.174377+0000","last_became_peered":"2020-12-28T10:29:18.174377+0000","last_unstale":"2020-12-28T10:29:58.110706+0000","last_undegraded":"2020-12-28T10:29:58.110706+0000","last_fullsized":"2020-12-28T10:29:58.110706+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:26:53.017342+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":"82","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:53.911337+0000","last_change":"2020-12-28T10:29:26.600978+0000","last_active":"2020-12-28T10:29:53.911337+0000","last_peered":"2020-12-28T10:29:53.911337+0000","last_clean":"2020-12-28T10:29:53.911337+0000","last_became_active":"2020-12-28T10:29:26.600150+0000","last_became_peered":"2020-12-28T10:29:26.600150+0000","last_unstale":"2020-12-28T10:29:53.911337+0000","last_undegraded":"2020-12-28T10:29:53.911337+0000","last_fullsized":"2020-12-28T10:29:53.911337+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:28:13.121978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:28:13.121978+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":4,"stat_sum":{"num_bytes":11723178816,"num_objects":3941,"num_object_clones":0,"num_object_copies":11823,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1954,"num_whiteouts":3,"num_read":11198,"num_read_kb":5604,"num_write":26944,"num_write_kb":7418970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":116994187,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":21,"num_bytes_hit_set_archive":12303,"num_flush":3643,"num_flush_kb":10623124,"num_evict":1682,"num_evict_kb":4889582,"num_promote":5602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15498022912,"data_stored":15381144729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12004,"ondisk_log_size":12004,"up":12,"acting":10,"num_store_stats":7},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":10867886122,"num_objects":3640,"num_object_clones":0,"num_object_copies":14560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3643,"num_write_kb":10623124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":742,"num_bytes_recovered":2201625494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15880802304,"data_stored":21765849088,"data_compressed":602711681,"data_compressed_allocated":5885046784,"data_compressed_original":11770093568,"omap_allocated":0,"internal_metadata":0},"log_size":3643,"ondisk_log_size":3643,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":2,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22,"ondisk_log_size":22,"up":33,"acting":33,"num_store_stats":8},{"poolid":2,"num_pg":18,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":54,"acting":54,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509482,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2667100,"kb_used_data":2607572,"kb_used_omap":0,"kb_used_meta":59520,"kb_avail":91704740,"statfs":{"total":96636764160,"available":93905653760,"internally_reserved":0,"allocated":2670153728,"data_stored":2649501886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":60948480},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 28 10:29:52 2020","interfaces":[{"interface":"back","average":{"1min":1.919,"5min":1.9530000000000001,"15min":2.0510000000000002},"min":{"1min":0.73599999999999999,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":6.8380000000000001,"5min":6.9960000000000004,"15min":6.9960000000000004},"last":2.0369999999999999},{"interface":"front","average":{"1min":1.841,"5min":2.105,"15min":2.2370000000000001},"min":{"1min":0.83199999999999996,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":6.3339999999999996,"5min":6.3339999999999996,"15min":6.3339999999999996},"last":2.1349999999999998}]},{"osd":1,"last update":"Mon Dec 28 10:29:52 2020","interfaces":[{"interface":"back","average":{"1min":2.1669999999999998,"5min":2.0800000000000001,"15min":2.1520000000000001},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":6.9279999999999999,"5min":6.9279999999999999,"15min":6.9279999999999999},"last":1.593},{"interface":"front","average":{"1min":2.2589999999999999,"5min":2.0840000000000001,"15min":2.1339999999999999},"min":{"1min":1.1220000000000001,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":5.4119999999999999,"5min":5.4119999999999999,"15min":5.4119999999999999},"last":1.6200000000000001}]},{"osd":2,"last update":"Mon Dec 28 10:29:52 2020","interfaces":[{"interface":"back","average":{"1min":2.2360000000000002,"5min":41.134,"15min":58.509999999999998},"min":{"1min":0.97699999999999998,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":6.4710000000000001,"5min":980.25199999999995,"15min":980.25199999999995},"last":2.1970000000000001},{"interface":"front","average":{"1min":2.4319999999999999,"5min":2.2490000000000001,"15min":2.3170000000000002},"min":{"1min":1.004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":6.5549999999999997,"5min":6.6230000000000002,"15min":6.6230000000000002},"last":2.266}]},{"osd":3,"last update":"Mon Dec 28 10:29:52 2020","interfaces":[{"interface":"back","average":{"1min":2.4039999999999999,"5min":11.791,"15min":5.5289999999999999},"min":{"1min":0.88600000000000001,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":7.0049999999999999,"5min":814.26099999999997,"15min":814.26099999999997},"last":1.337},{"interface":"front","average":{"1min":2.5470000000000002,"5min":2.395,"15min":2.403},"min":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"max":{"1min":7.2409999999999997,"5min":7.2409999999999997,"15min":7.2409999999999997},"last":1.736}]},{"osd":4,"last update":"Mon Dec 28 10:27:48 2020","interfaces":[{"interface":"back","average":{"1min":2.875,"5min":2.875,"15min":2.875},"min":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"max":{"1min":6.6619999999999999,"5min":6.6619999999999999,"15min":6.6619999999999999},"last":1.6890000000000001},{"interface":"front","average":{"1min":2.6819999999999999,"5min":2.6819999999999999,"15min":2.6819999999999999},"min":{"1min":1.216,"5min":1.216,"15min":1.216},"max":{"1min":3.5670000000000002,"5min":3.5670000000000002,"15min":3.5670000000000002},"last":1.0069999999999999}]},{"osd":5,"last update":"Mon Dec 28 10:29:52 2020","interfaces":[{"interface":"back","average":{"1min":13.756,"5min":21.114000000000001,"15min":26.917999999999999},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"max":{"1min":173.16499999999999,"5min":410.97399999999999,"15min":410.97399999999999},"last":1.7929999999999999},{"interface":"front","average":{"1min":13.831,"5min":16.829999999999998,"15min":20.641999999999999},"min":{"1min":1.141,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":175.053,"5min":292.35399999999998,"15min":292.35399999999998},"last":1.359}]},{"osd":6,"last update":"Mon Dec 28 10:29:52 2020","interfaces":[{"interface":"back","average":{"1min":2.742,"5min":2.98,"15min":3.25},"min":{"1min":1.044,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":6.7690000000000001,"5min":11.333,"15min":11.333},"last":1.4330000000000001},{"interface":"front","average":{"1min":43.289000000000001,"5min":10.798,"15min":5.6079999999999997},"min":{"1min":1.244,"5min":1.1040000000000001,"15min":1.1040000000000001},"max":{"1min":768.08699999999999,"5min":768.08699999999999,"15min":768.08699999999999},"last":1.3939999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509481,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4253836,"kb_used_data":4198276,"kb_used_omap":0,"kb_used_meta":55552,"kb_avail":90118004,"statfs":{"total":96636764160,"available":92280836096,"internally_reserved":0,"allocated":4299034624,"data_stored":5377428259,"data_compressed":111428798,"data_compressed_allocated":1088954368,"data_compressed_original":2177908736,"omap_allocated":0,"internal_metadata":56885248},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 28 10:29:56 2020","interfaces":[{"interface":"back","average":{"1min":1.859,"5min":42.652999999999999,"15min":51.728999999999999},"min":{"1min":0.91100000000000003,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":5.5620000000000003,"5min":981.12800000000004,"15min":981.12800000000004},"last":2.5369999999999999},{"interface":"front","average":{"1min":36,"5min":8.7400000000000002,"15min":4.2329999999999997},"min":{"1min":1.228,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":612.89800000000002,"5min":612.89800000000002,"15min":612.89800000000002},"last":612.89800000000002}]},{"osd":1,"last update":"Mon Dec 28 10:29:56 2020","interfaces":[{"interface":"back","average":{"1min":2.298,"5min":1.994,"15min":1.8600000000000001},"min":{"1min":0.84199999999999997,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":6.657,"5min":8.032,"15min":8.032},"last":1.7649999999999999},{"interface":"front","average":{"1min":46.810000000000002,"5min":43.213000000000001,"15min":41.155000000000001},"min":{"1min":0.90500000000000003,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":809.19899999999996,"5min":849.45399999999995,"15min":849.45399999999995},"last":2.851}]},{"osd":2,"last update":"Mon Dec 28 10:29:56 2020","interfaces":[{"interface":"back","average":{"1min":2.3450000000000002,"5min":2.254,"15min":2.306},"min":{"1min":0.93999999999999995,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":3.9260000000000002,"5min":5.7409999999999997,"15min":5.7409999999999997},"last":2.988},{"interface":"front","average":{"1min":2.0870000000000002,"5min":2.1699999999999999,"15min":2.2799999999999998},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":4.2229999999999999,"5min":5.5300000000000002,"15min":5.5300000000000002},"last":2.403}]},{"osd":3,"last update":"Mon Dec 28 10:29:56 2020","interfaces":[{"interface":"back","average":{"1min":45.154000000000003,"5min":13.334,"15min":8.5139999999999993},"min":{"1min":1.6830000000000001,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":770.32899999999995,"5min":770.32899999999995,"15min":770.32899999999995},"last":3.0699999999999998},{"interface":"front","average":{"1min":2.7829999999999999,"5min":4.2320000000000002,"15min":4.9409999999999998},"min":{"1min":1.571,"5min":0.98999999999999999,"15min":0.98999999999999999},"max":{"1min":6.7859999999999996,"5min":61.508000000000003,"15min":61.508000000000003},"last":2.4609999999999999}]},{"osd":4,"last update":"Mon Dec 28 10:27:50 2020","interfaces":[{"interface":"back","average":{"1min":2.4359999999999999,"5min":2.4359999999999999,"15min":2.4359999999999999},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":3.6629999999999998,"5min":3.6629999999999998,"15min":3.6629999999999998},"last":2.2919999999999998},{"interface":"front","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":8.2240000000000002,"5min":8.2240000000000002,"15min":8.2240000000000002},"last":1.8420000000000001}]},{"osd":5,"last update":"Mon Dec 28 10:29:56 2020","interfaces":[{"interface":"back","average":{"1min":12.115,"5min":4.5999999999999996,"15min":3.3410000000000002},"min":{"1min":1.1180000000000001,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":175.49700000000001,"5min":175.49700000000001,"15min":175.49700000000001},"last":2.6080000000000001},{"interface":"front","average":{"1min":11.923,"5min":4.5570000000000004,"15min":3.2669999999999999},"min":{"1min":1.0940000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"max":{"1min":176.26400000000001,"5min":176.26400000000001,"15min":176.26400000000001},"last":3.1469999999999998}]},{"osd":7,"last update":"Mon Dec 28 10:29:56 2020","interfaces":[{"interface":"back","average":{"1min":2.585,"5min":18.681000000000001,"15min":8.2390000000000008},"min":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"max":{"1min":6.4009999999999998,"5min":856.17700000000002,"15min":856.17700000000002},"last":2.681},{"interface":"front","average":{"1min":3.0579999999999998,"5min":12.347,"15min":6.0389999999999997},"min":{"1min":1.026,"5min":0.94899999999999995,"15min":0.94899999999999995},"max":{"1min":9.9260000000000002,"5min":503.36900000000003,"15min":503.36900000000003},"last":2.9239999999999999}]}]},{"osd":1,"up_from":15,"seq":64424509482,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2325524,"kb_used_data":2284812,"kb_used_omap":0,"kb_used_meta":40704,"kb_avail":92046316,"statfs":{"total":96636764160,"available":94255427584,"internally_reserved":0,"allocated":2339647488,"data_stored":2696888154,"data_compressed":37719632,"data_compressed_allocated":368033792,"data_compressed_original":736067584,"omap_allocated":0,"internal_metadata":41680896},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 28 10:29:53 2020","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.7350000000000001,"15min":1.8100000000000001},"min":{"1min":0.51300000000000001,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":2.7189999999999999,"5min":4.0819999999999999,"15min":4.0819999999999999},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.7430000000000001,"5min":2.2679999999999998,"15min":2.444},"min":{"1min":0.59399999999999997,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":2.9830000000000001,"5min":10.836,"15min":10.836},"last":2.9830000000000001}]},{"osd":2,"last update":"Mon Dec 28 10:29:53 2020","interfaces":[{"interface":"back","average":{"1min":1.756,"5min":12.887,"15min":5.359},"min":{"1min":0.66200000000000003,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":6.29,"5min":784.22199999999998,"15min":784.22199999999998},"last":1.538},{"interface":"front","average":{"1min":1.835,"5min":1.9590000000000001,"15min":1.9650000000000001},"min":{"1min":0.64600000000000002,"5min":0.625,"15min":0.625},"max":{"1min":3.819,"5min":12.396000000000001,"15min":12.396000000000001},"last":1.403}]},{"osd":3,"last update":"Mon Dec 28 10:29:53 2020","interfaces":[{"interface":"back","average":{"1min":61.206000000000003,"5min":40.906999999999996,"15min":45.015000000000001},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":507.279,"5min":753.41999999999996,"15min":753.41999999999996},"last":1.946},{"interface":"front","average":{"1min":41.82,"5min":10.255000000000001,"15min":5.0549999999999997},"min":{"1min":0.93200000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":507.13,"5min":507.13,"15min":507.13},"last":2.7759999999999998}]},{"osd":4,"last update":"Mon Dec 28 10:27:48 2020","interfaces":[{"interface":"back","average":{"1min":2.137,"5min":2.137,"15min":2.137},"min":{"1min":1.03,"5min":1.03,"15min":1.03},"max":{"1min":4.6180000000000003,"5min":4.6180000000000003,"15min":4.6180000000000003},"last":1.609},{"interface":"front","average":{"1min":2.3889999999999998,"5min":2.3889999999999998,"15min":2.3889999999999998},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":4.3860000000000001,"5min":4.3860000000000001,"15min":4.3860000000000001},"last":1.859}]},{"osd":5,"last update":"Mon Dec 28 10:29:53 2020","interfaces":[{"interface":"back","average":{"1min":3.3170000000000002,"5min":3.2160000000000002,"15min":3.2989999999999999},"min":{"1min":1.016,"5min":1.016,"15min":1.016},"max":{"1min":12.57,"5min":12.57,"15min":12.57},"last":1.665},{"interface":"front","average":{"1min":2.669,"5min":2.4340000000000002,"15min":2.3639999999999999},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":4.7140000000000004,"5min":11.34,"15min":11.34},"last":2.8460000000000001}]},{"osd":6,"last update":"Mon Dec 28 10:29:53 2020","interfaces":[{"interface":"back","average":{"1min":2.121,"5min":2.4420000000000002,"15min":2.5779999999999998},"min":{"1min":0.95699999999999996,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":4.0250000000000004,"5min":4.2519999999999998,"15min":4.2519999999999998},"last":2.028},{"interface":"front","average":{"1min":18.768000000000001,"5min":15.351000000000001,"15min":7.5529999999999999},"min":{"1min":1.131,"5min":1.0329999999999999,"15min":1.0329999999999999},"max":{"1min":344.22500000000002,"5min":757.19799999999998,"15min":757.19799999999998},"last":2.9119999999999999}]},{"osd":7,"last update":"Mon Dec 28 10:29:53 2020","interfaces":[{"interface":"back","average":{"1min":2.6160000000000001,"5min":3.0619999999999998,"15min":3.254},"min":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"max":{"1min":4.2939999999999996,"5min":8.8450000000000006,"15min":8.8450000000000006},"last":3.04},{"interface":"front","average":{"1min":2.5859999999999999,"5min":2.4990000000000001,"15min":2.46},"min":{"1min":1.2230000000000001,"5min":0.96599999999999997,"15min":0.96599999999999997},"max":{"1min":6.2569999999999997,"5min":11.372,"15min":11.372},"last":2.3860000000000001}]}]},{"osd":0,"up_from":15,"seq":64424509481,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4600336,"kb_used_data":4538568,"kb_used_omap":0,"kb_used_meta":61760,"kb_avail":89771504,"statfs":{"total":96636764160,"available":91926020096,"internally_reserved":0,"allocated":4647493632,"data_stored":5358985406,"data_compressed":75025437,"data_compressed_allocated":732143616,"data_compressed_original":1464287232,"omap_allocated":0,"internal_metadata":63242240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Dec 28 10:29:58 2020","interfaces":[{"interface":"back","average":{"1min":1.679,"5min":1.5649999999999999,"15min":1.548},"min":{"1min":0.65900000000000003,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":3.5800000000000001,"5min":3.9430000000000001,"15min":3.9430000000000001},"last":1.821},{"interface":"front","average":{"1min":1.9630000000000001,"5min":1.8089999999999999,"15min":1.653},"min":{"1min":0.88,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":3.3199999999999998,"5min":17.34,"15min":17.34},"last":1.8540000000000001}]},{"osd":2,"last update":"Mon Dec 28 10:29:58 2020","interfaces":[{"interface":"back","average":{"1min":46.095999999999997,"5min":28.902999999999999,"15min":31.105},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":527.29200000000003,"5min":527.29200000000003,"15min":527.29200000000003},"last":2.4449999999999998},{"interface":"front","average":{"1min":2.0059999999999998,"5min":1.831,"15min":1.7330000000000001},"min":{"1min":1.175,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":3.2250000000000001,"5min":8.7400000000000002,"15min":8.7400000000000002},"last":1.175}]},{"osd":3,"last update":"Mon Dec 28 10:29:58 2020","interfaces":[{"interface":"back","average":{"1min":51.545000000000002,"5min":11.798,"15min":5.1520000000000001},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":983.34400000000005,"5min":983.34400000000005,"15min":983.34400000000005},"last":2.8849999999999998},{"interface":"front","average":{"1min":12.106,"5min":4.4550000000000001,"15min":2.9079999999999999},"min":{"1min":1.103,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":187.756,"5min":187.756,"15min":187.756},"last":3.056}]},{"osd":4,"last update":"Mon Dec 28 10:27:52 2020","interfaces":[{"interface":"back","average":{"1min":1.974,"5min":1.974,"15min":1.974},"min":{"1min":1.095,"5min":1.095,"15min":1.095},"max":{"1min":2.8450000000000002,"5min":2.8450000000000002,"15min":2.8450000000000002},"last":7.1790000000000003},{"interface":"front","average":{"1min":2.3530000000000002,"5min":2.3530000000000002,"15min":2.3530000000000002},"min":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"max":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"last":2.1579999999999999}]},{"osd":5,"last update":"Mon Dec 28 10:29:58 2020","interfaces":[{"interface":"back","average":{"1min":2.8330000000000002,"5min":5.1509999999999998,"15min":3.1349999999999998},"min":{"1min":1.385,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":5.9370000000000003,"5min":355.34500000000003,"15min":355.34500000000003},"last":3.2749999999999999},{"interface":"front","average":{"1min":2.7669999999999999,"5min":2.5899999999999999,"15min":2.3359999999999999},"min":{"1min":1.1080000000000001,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":9.8320000000000007,"5min":28.140000000000001,"15min":28.140000000000001},"last":1.6060000000000001}]},{"osd":6,"last update":"Mon Dec 28 10:29:58 2020","interfaces":[{"interface":"back","average":{"1min":2.4249999999999998,"5min":3.4740000000000002,"15min":2.8319999999999999},"min":{"1min":1.1319999999999999,"5min":0.88800000000000001,"15min":0.88800000000000001},"max":{"1min":3.9159999999999999,"5min":134.11600000000001,"15min":134.11600000000001},"last":2.419},{"interface":"front","average":{"1min":18.087,"5min":6.665,"15min":3.9359999999999999},"min":{"1min":1.73,"5min":0.95299999999999996,"15min":0.95299999999999996},"max":{"1min":225.87899999999999,"5min":225.87899999999999,"15min":225.87899999999999},"last":2.988}]},{"osd":7,"last update":"Mon Dec 28 10:29:58 2020","interfaces":[{"interface":"back","average":{"1min":2.754,"5min":2.472,"15min":2.4399999999999999},"min":{"1min":1.1259999999999999,"5min":1.008,"15min":1.008},"max":{"1min":4.5140000000000002,"5min":8.8529999999999998,"15min":8.8529999999999998},"last":2.6030000000000002},{"interface":"front","average":{"1min":2.3010000000000002,"5min":15.318,"15min":21.201000000000001},"min":{"1min":1.3,"5min":1.0840000000000001,"15min":1.0840000000000001},"max":{"1min":3.3919999999999999,"5min":371.392,"15min":371.392},"last":2.2269999999999999}]}]},{"osd":2,"up_from":15,"seq":64424509481,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1992600,"kb_used_data":1963984,"kb_used_omap":0,"kb_used_meta":28608,"kb_avail":92379240,"statfs":{"total":96636764160,"available":94596341760,"internally_reserved":0,"allocated":2011119616,"data_stored":2756233773,"data_compressed":76448272,"data_compressed_allocated":745623552,"data_compressed_original":1491247104,"omap_allocated":0,"internal_metadata":29294592},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 28 10:29:52 2020","interfaces":[{"interface":"back","average":{"1min":1.923,"5min":6.8739999999999997,"15min":8.9039999999999999},"min":{"1min":0.73799999999999999,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":3.6480000000000001,"5min":162.44499999999999,"15min":162.44499999999999},"last":1.4910000000000001},{"interface":"front","average":{"1min":2.1030000000000002,"5min":4.25,"15min":5.1200000000000001},"min":{"1min":0.78200000000000003,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":3.9620000000000002,"5min":76.317999999999998,"15min":76.317999999999998},"last":2.367}]},{"osd":1,"last update":"Mon Dec 28 10:29:52 2020","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":1.855,"15min":1.9610000000000001},"min":{"1min":0.65900000000000003,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":3.1739999999999999,"5min":5.6660000000000004,"15min":5.6660000000000004},"last":1.071},{"interface":"front","average":{"1min":2.0800000000000001,"5min":5.2389999999999999,"15min":3.1230000000000002},"min":{"1min":1.1519999999999999,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":4.3529999999999998,"5min":281.38299999999998,"15min":281.38299999999998},"last":1.5820000000000001}]},{"osd":3,"last update":"Mon Dec 28 10:29:55 2020","interfaces":[{"interface":"back","average":{"1min":1.9099999999999999,"5min":1.8149999999999999,"15min":1.7130000000000001},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":3.6579999999999999,"5min":4.6449999999999996,"15min":4.6449999999999996},"last":1.631},{"interface":"front","average":{"1min":1.806,"5min":2.0630000000000002,"15min":2.1629999999999998},"min":{"1min":0.85499999999999998,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":3.0960000000000001,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":1.0009999999999999}]},{"osd":4,"last update":"Mon Dec 28 10:27:51 2020","interfaces":[{"interface":"back","average":{"1min":2.3250000000000002,"5min":2.3250000000000002,"15min":2.3250000000000002},"min":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"max":{"1min":5.6980000000000004,"5min":5.6980000000000004,"15min":5.6980000000000004},"last":2.3210000000000002},{"interface":"front","average":{"1min":2.3069999999999999,"5min":2.3069999999999999,"15min":2.3069999999999999},"min":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"max":{"1min":4.0730000000000004,"5min":4.0730000000000004,"15min":4.0730000000000004},"last":1.554}]},{"osd":5,"last update":"Mon Dec 28 10:29:52 2020","interfaces":[{"interface":"back","average":{"1min":23.582000000000001,"5min":6.5010000000000003,"15min":3.6800000000000002},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":365.40800000000002,"5min":365.40800000000002,"15min":365.40800000000002},"last":1.6819999999999999},{"interface":"front","average":{"1min":2.29,"5min":26.539999999999999,"15min":37.118000000000002},"min":{"1min":1.1910000000000001,"5min":1.028,"15min":1.028},"max":{"1min":4.8410000000000002,"5min":758.39099999999996,"15min":758.39099999999996},"last":1.7430000000000001}]},{"osd":6,"last update":"Mon Dec 28 10:29:52 2020","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.4359999999999999,"15min":2.5019999999999998},"min":{"1min":1.2929999999999999,"5min":0.95999999999999996,"15min":0.95999999999999996},"max":{"1min":4.0510000000000002,"5min":6.5700000000000003,"15min":6.5700000000000003},"last":2.2709999999999999},{"interface":"front","average":{"1min":2.4390000000000001,"5min":2.4239999999999999,"15min":2.4769999999999999},"min":{"1min":1.2450000000000001,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":4.359,"5min":6.4640000000000004,"15min":6.4640000000000004},"last":2.4089999999999998}]},{"osd":7,"last update":"Mon Dec 28 10:29:52 2020","interfaces":[{"interface":"back","average":{"1min":2.6859999999999999,"5min":2.7229999999999999,"15min":2.8559999999999999},"min":{"1min":1.637,"5min":1.002,"15min":1.002},"max":{"1min":4.4889999999999999,"5min":6.9870000000000001,"15min":6.9870000000000001},"last":2.448},{"interface":"front","average":{"1min":2.48,"5min":2.891,"15min":3.145},"min":{"1min":1.2869999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"max":{"1min":4.7290000000000001,"5min":6.8979999999999997,"15min":6.8979999999999997},"last":1.788}]}]},{"osd":3,"up_from":15,"seq":64424509481,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3321044,"kb_used_data":3279436,"kb_used_omap":0,"kb_used_meta":41600,"kb_avail":91050796,"statfs":{"total":96636764160,"available":93236015104,"internally_reserved":0,"allocated":3358142464,"data_stored":4099801574,"data_compressed":77073221,"data_compressed_allocated":752254976,"data_compressed_original":1504509952,"omap_allocated":0,"internal_metadata":42598400},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 28 10:29:57 2020","interfaces":[{"interface":"back","average":{"1min":2.0609999999999999,"5min":35.347000000000001,"15min":46.389000000000003},"min":{"1min":0.68200000000000005,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":4.0960000000000001,"5min":953.36000000000001,"15min":953.36000000000001},"last":4.0960000000000001},{"interface":"front","average":{"1min":1.665,"5min":1.8280000000000001,"15min":1.6819999999999999},"min":{"1min":0.67700000000000005,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":4.0780000000000003,"5min":12.409000000000001,"15min":12.409000000000001},"last":1.849}]},{"osd":1,"last update":"Mon Dec 28 10:29:57 2020","interfaces":[{"interface":"back","average":{"1min":1.7110000000000001,"5min":1.861,"15min":1.831},"min":{"1min":0.59299999999999997,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":2.7269999999999999,"5min":6.9219999999999997,"15min":6.9219999999999997},"last":2.5790000000000002},{"interface":"front","average":{"1min":2.2839999999999998,"5min":2.0710000000000002,"15min":2.085},"min":{"1min":1.0029999999999999,"5min":0.629,"15min":0.629},"max":{"1min":4.1630000000000003,"5min":6.3739999999999997,"15min":6.3739999999999997},"last":3.1419999999999999}]},{"osd":2,"last update":"Mon Dec 28 10:29:57 2020","interfaces":[{"interface":"back","average":{"1min":2.0800000000000001,"5min":2.0899999999999999,"15min":2.1000000000000001},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":4.0220000000000002,"5min":5.7800000000000002,"15min":5.7800000000000002},"last":3.8210000000000002},{"interface":"front","average":{"1min":2.7040000000000002,"5min":2.2440000000000002,"15min":2.27},"min":{"1min":1.0429999999999999,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":4.117,"5min":4.8890000000000002,"15min":4.8890000000000002},"last":3.2040000000000002}]},{"osd":4,"last update":"Mon Dec 28 10:27:50 2020","interfaces":[{"interface":"back","average":{"1min":2.6549999999999998,"5min":2.6549999999999998,"15min":2.6549999999999998},"min":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"max":{"1min":5.1959999999999997,"5min":5.1959999999999997,"15min":5.1959999999999997},"last":2.8359999999999999},{"interface":"front","average":{"1min":2.7509999999999999,"5min":2.7509999999999999,"15min":2.7509999999999999},"min":{"1min":1.262,"5min":1.262,"15min":1.262},"max":{"1min":4.7080000000000002,"5min":4.7080000000000002,"15min":4.7080000000000002},"last":2.9950000000000001}]},{"osd":5,"last update":"Mon Dec 28 10:29:57 2020","interfaces":[{"interface":"back","average":{"1min":2.3879999999999999,"5min":2.806,"15min":2.883},"min":{"1min":0.93799999999999994,"5min":0.874,"15min":0.874},"max":{"1min":4.1550000000000002,"5min":10.362,"15min":10.362},"last":4.1550000000000002},{"interface":"front","average":{"1min":2.9710000000000001,"5min":2.887,"15min":2.8620000000000001},"min":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"max":{"1min":4.9580000000000002,"5min":11.971,"15min":11.971},"last":3.0790000000000002}]},{"osd":6,"last update":"Mon Dec 28 10:29:57 2020","interfaces":[{"interface":"back","average":{"1min":13.807,"5min":13.57,"15min":6.4790000000000001},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":180.655,"5min":813.024,"15min":813.024},"last":3.9910000000000001},{"interface":"front","average":{"1min":2.5070000000000001,"5min":2.6349999999999998,"15min":2.7290000000000001},"min":{"1min":1.044,"5min":1,"15min":1},"max":{"1min":3.9220000000000002,"5min":6.2750000000000004,"15min":6.2750000000000004},"last":3.9220000000000002}]},{"osd":7,"last update":"Mon Dec 28 10:29:57 2020","interfaces":[{"interface":"back","average":{"1min":3.052,"5min":2.7029999999999998,"15min":2.7749999999999999},"min":{"1min":1.3149999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"max":{"1min":5.0229999999999997,"5min":5.984,"15min":5.984},"last":3.3119999999999998},{"interface":"front","average":{"1min":2.778,"5min":2.8980000000000001,"15min":3.056},"min":{"1min":1.075,"5min":1.075,"15min":1.075},"max":{"1min":4.6769999999999996,"5min":6.2729999999999997,"15min":6.2729999999999997},"last":4.6769999999999996}]}]},{"osd":4,"up_from":94,"seq":403726925835,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3974984,"kb_used_data":3926440,"kb_used_omap":0,"kb_used_meta":37631,"kb_avail":90396856,"statfs":{"total":96636764160,"available":92566380544,"internally_reserved":0,"allocated":4020674560,"data_stored":4730188751,"data_compressed":74266497,"data_compressed_allocated":725573632,"data_compressed_original":1451147264,"omap_allocated":964,"internal_metadata":38534204},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.204}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.484}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.53}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6020000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0819999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.6219999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0470000000000002}]}]},{"osd":5,"up_from":15,"seq":64424509482,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7909896,"kb_used_data":7815680,"kb_used_omap":0,"kb_used_meta":94208,"kb_avail":86461944,"statfs":{"total":96636764160,"available":88537030656,"internally_reserved":0,"allocated":8003256320,"data_stored":9444818699,"data_compressed":150749824,"data_compressed_allocated":1472462848,"data_compressed_original":2944925696,"omap_allocated":0,"internal_metadata":96468992},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 28 10:29:53 2020","interfaces":[{"interface":"back","average":{"1min":2.0630000000000002,"5min":4.2270000000000003,"15min":2.8220000000000001},"min":{"1min":0.72299999999999998,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":4.4500000000000002,"5min":168.51599999999999,"15min":168.51599999999999},"last":3.1419999999999999},{"interface":"front","average":{"1min":2.3260000000000001,"5min":4.2729999999999997,"15min":2.8039999999999998},"min":{"1min":0.76700000000000002,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":4.4880000000000004,"5min":168.56200000000001,"15min":168.56200000000001},"last":3.2629999999999999}]},{"osd":1,"last update":"Mon Dec 28 10:29:53 2020","interfaces":[{"interface":"back","average":{"1min":31.669,"5min":7.9210000000000003,"15min":3.774},"min":{"1min":1.0780000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":504.11000000000001,"5min":504.11000000000001,"15min":504.11000000000001},"last":3.0680000000000001},{"interface":"front","average":{"1min":31.68,"5min":18.266999999999999,"15min":7.3570000000000002},"min":{"1min":0.79900000000000004,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":504.06900000000002,"5min":870.06200000000001,"15min":870.06200000000001},"last":1.6819999999999999}]},{"osd":2,"last update":"Mon Dec 28 10:29:53 2020","interfaces":[{"interface":"back","average":{"1min":87.649000000000001,"5min":19.332000000000001,"15min":7.8899999999999997},"min":{"1min":0.87,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":985.255,"5min":985.255,"15min":985.255},"last":1.4990000000000001},{"interface":"front","average":{"1min":3.02,"5min":2.4769999999999999,"15min":2.2909999999999999},"min":{"1min":1.2210000000000001,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":13.564,"5min":13.564,"15min":13.564},"last":1.6240000000000001}]},{"osd":3,"last update":"Mon Dec 28 10:29:53 2020","interfaces":[{"interface":"back","average":{"1min":2.3679999999999999,"5min":2.3719999999999999,"15min":2.254},"min":{"1min":1.014,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":3.7770000000000001,"5min":11.653,"15min":11.653},"last":3.3239999999999998},{"interface":"front","average":{"1min":2.6800000000000002,"5min":2.665,"15min":2.5609999999999999},"min":{"1min":1.089,"5min":0.90000000000000002,"15min":0.90000000000000002},"max":{"1min":8.8889999999999993,"5min":10.91,"15min":10.91},"last":2.0499999999999998}]},{"osd":4,"last update":"Mon Dec 28 10:27:50 2020","interfaces":[{"interface":"back","average":{"1min":2.726,"5min":2.726,"15min":2.726},"min":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"max":{"1min":5.601,"5min":5.601,"15min":5.601},"last":3.2000000000000002},{"interface":"front","average":{"1min":2.5390000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"min":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"max":{"1min":5.8019999999999996,"5min":5.8019999999999996,"15min":5.8019999999999996},"last":2.0049999999999999}]},{"osd":6,"last update":"Mon Dec 28 10:29:53 2020","interfaces":[{"interface":"back","average":{"1min":2.5449999999999999,"5min":2.7839999999999998,"15min":2.7709999999999999},"min":{"1min":1.0449999999999999,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":10.602,"5min":13.284000000000001,"15min":13.284000000000001},"last":1.5720000000000001},{"interface":"front","average":{"1min":3.2909999999999999,"5min":6.8470000000000004,"15min":4.032},"min":{"1min":1.1339999999999999,"5min":1.044,"15min":1.044},"max":{"1min":13.528,"5min":335.69,"15min":335.69},"last":1.9670000000000001}]},{"osd":7,"last update":"Mon Dec 28 10:28:58 2020","interfaces":[{"interface":"back","average":{"1min":3.1899999999999999,"5min":8.5540000000000003,"15min":9.4480000000000004},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":11.612,"5min":163.16399999999999,"15min":163.16399999999999},"last":3.3799999999999999},{"interface":"front","average":{"1min":42.051000000000002,"5min":10.766,"15min":5.5519999999999996},"min":{"1min":1.2370000000000001,"5min":1.069,"15min":1.069},"max":{"1min":661.77800000000002,"5min":661.77800000000002,"15min":661.77800000000002},"last":2.1019999999999999}]}]}]}} , 2020-12-28T10:30:05.024+0000 7f108c9d0700 -1 osd.3 pg_epoch: 120 pg[3.2s0( v 91'925 (0'0,91'925] local-lis/les=118/119 n=923 ec=23/23 lis/c=118/118 les/c/f=119/119/0 sis=118) [3,5,6,4]p3(0) r=0 lpr=118 crt=91'925 lcod 91'924 mlcod 0'0 active+clean] scrubber pg(3.2s0) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6(2) (0x560d16fbf5a0) Ignoring malparsed backtrace: 2020-12-28T10:30:35.025 INFO:tasks.ceph.ceph_manager.ceph:recovered! , 2020-12-28T10:30:35.026 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes , 2020-12-28T10:30:35.356 DEBUG:teuthology.run_tasks:Unwinding manager ceph Ignoring malparsed backtrace: 2020-12-28T10:30:41.751 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2020-12-28T10:28:14.506230+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=28, tm_hour=10, tm_min=28, tm_sec=14, tm_wday=0, tm_yday=363, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=28, tm_hour=10, tm_min=30, tm_sec=36, tm_wday=0, tm_yday=363, tm_isdst=0) , 2020-12-28T10:30:41.752 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2020-12-28T10:28:38.540092+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=28, tm_hour=10, tm_min=28, tm_sec=38, tm_wday=0, tm_yday=363, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=28, tm_hour=10, tm_min=30, tm_sec=36, tm_wday=0, tm_yday=363, tm_isdst=0) , 2020-12-28T10:30:41.752 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2020-12-28T10:30:24.640833+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=28, tm_hour=10, tm_min=30, tm_sec=24, tm_wday=0, tm_yday=363, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=28, tm_hour=10, tm_min=30, tm_sec=36, tm_wday=0, tm_yday=363, tm_isdst=0) Ignoring malparsed backtrace: 2020-12-28T10:32:01.076+0000 7fbf94d65700 -1 osd.0 pg_epoch: 134 pg[4.3( v 130'8038 (78'5037,130'8038] local-lis/les=26/27 n=986 ec=26/26 lis/c=26/26 les/c/f=27/27/0 sis=26) [0,5,7] r=0 lpr=26 crt=130'8038 lcod 130'8037 mlcod 130'8037 active+clean] scrubber pg(4.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55b922c946e0) , 2020-12-28T10:32:02.048+0000 7fbf90d5d700 -1 osd.0 pg_epoch: 134 pg[4.3( v 130'8038 (78'5037,130'8038] local-lis/les=26/27 n=986 ec=26/26 lis/c=26/26 les/c/f=27/27/0 sis=26) [0,5,7] r=0 lpr=26 crt=130'8038 lcod 130'8037 mlcod 130'8037 active+clean] scrubber pg(4.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55b92bd01ce0) , 2020-12-28T10:32:02.120+0000 7f44eeebf700 -1 osd.2 pg_epoch: 134 pg[2.10( empty local-lis/les=102/103 n=0 ec=63/18 lis/c=102/102 les/c/f=103/103/0 sis=102) [2,1,0] r=0 lpr=102 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.10) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x556272404c60) Ignoring malparsed backtrace: 2020-12-28T10:32:52.019 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi092.front.sepia.ceph.com , 2020-12-28T10:32:52.019 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2020-12-28T10:32:52.218 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored Ignoring malparsed backtrace: 2020-12-28T10:37:41.506 INFO:teuthology.orchestra.run.smithi016.stdout:============================================================================== , 2020-12-28T10:37:41.506 INFO:teuthology.orchestra.run.smithi016.stdout:-hv01.front.sepi 216.240.36.24 3 u 19 64 377 0.095 0.915 1.676 , 2020-12-28T10:37:41.506 INFO:teuthology.orchestra.run.smithi016.stdout:*hv02.front.sepi 163.237.218.18 2 u 21 64 377 0.095 0.059 1.985 Found 11 distinct failure reasons Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2 jobs: ['5742346', '5742190'] suites intersection: ['msgr/async', 'rados'] suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/cache-agent-big}'] Dead: b'2020-12-28T21:55:52.665 DEBUG:teuthology.exit:Finished running handlers' 6 jobs: ['5742162', '5742120', '5742156', '5742293', '5742212', '5742014'] suites intersection: ['thrashosds-health'] suites union: ['1-install/nautilus', '1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-partial-recovery}', '3-thrash/default', '4-workload/{rbd-cls', '5-workload/{radosbench', '6-finish-upgrade', '7-octopus', '8-workload/{rbd-python', 'backoff/peering_and_degraded', 'bluestore-bitmap', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-hybrid', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack', 'rbd-import-export', 'rbd_api}', 'readwrite', 'recovery-overrides/{default}', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashers/pggrow', 'thrashosds-health', 'ubuntu_latest}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/rados_api_tests}', 'workloads/radosbench-high-concurrency}', 'workloads/small-objects}', 'workloads/snaps-few-objects}'] Failure: Command failed (workunit test cephadm/test_adoption.sh) on smithi071 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ce2ba5cb8764cac5bd77397fd0c120a799451956 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' 1 jobs: ['5742080'] suites: ['mon_election/connectivity', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_adoption}'] Failure: unexpected end of data 1 jobs: ['5742043'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/none', 'thrashosds-health', 'workloads/radosbench}'] Failure: reached maximum tries (180) after waiting for 180 seconds 5 jobs: ['5742356', '5742172', '5742261', '5742084', '5742228'] suites intersection: ['3-start-upgrade', '4-wait', 'fixed-2', 'rados/cephadm/upgrade/{1-start'] suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro$/{rhel_8.0}', 'distro$/{rhel_latest}', 'fixed-2', 'mon_election/classic}', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start'] Timeout 6h running clone.client.0/qa/workunits/rados/test.sh 2 jobs: ['5742053', '5742272'] suites intersection: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}'] suites union: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}'] Failure: "2020-12-28T10:37:58.458616+0000 mon.a (mon.0) 21 : cluster [WRN] Health check failed: 1/9 mons down, quorum a,b,c,d,e,f,g,h (MON_DOWN)" in cluster log 1 jobs: ['5742201'] suites: ['mon_election/connectivity', 'msgr-failures/many', 'msgr/async-v1only', 'no_pools', 'objectstore/bluestore-low-osd-mem-target', 'rados', 'rados/multimon/{clusters/9', 'supported-random-distro$/{rhel_8}', 'tasks/mon_clock_no_skews}'] Failure: psutil.NoSuchProcess process no longer exists (pid=5756) 1 jobs: ['5742361'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/small-objects-balanced}'] Failure: Exiting scrub checking -- not all pgs scrubbed. 1 jobs: ['5742222'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-partial-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/cache-pool-snaps}'] Failure: Command failed (workunit test rados/test.sh) on smithi099 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ce2ba5cb8764cac5bd77397fd0c120a799451956 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 1 jobs: ['5742284'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/default', 'thrashosds-health', 'workloads/rados_api_tests}'] Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi080 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ce2ba5cb8764cac5bd77397fd0c120a799451956 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 1 jobs: ['5742254'] suites: ['mon_election/connectivity', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_cephadm}']