2020-11-18T10:16:24.370 INFO:root:teuthology version: 1.0.0-357fd4a 2020-11-18T10:16:24.371 INFO:teuthology.results:subset: '17/100000' 2020-11-18T10:16:24.371 INFO:teuthology.results:seed: '2420' 2020-11-18T13:24:39.030 INFO:root:teuthology version: 1.0.0-357fd4a 2020-11-18T13:24:39.458 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-11-18T18:41:50.744 INFO:teuthology.results:Tests finished! gathering results... 2020-11-18T18:42:34.873 INFO:scrape:Found 367 jobs 2020-11-18T18:42:47.179 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-7273-gfbf83bb3 (fbf83bb377e36caf429a163721492f1b1c08deb9) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f14e3a605d4] , 2: (DeleteOp::_begin()+0x519) [0x55ce262f4099] 2020-11-18T18:42:47.181 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-18T12:02:21.315 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2020-11-18T12:02:20.101473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29907025880,"num_objects":10021,"num_object_clones":0,"num_object_copies":20042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9996,"num_whiteouts":0,"num_read":21560,"num_read_kb":111566,"num_write":52242,"num_write_kb":23304041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":25,"num_bytes_hit_set_archive":12609,"num_flush":3426,"num_flush_kb":10013824,"num_evict":3017,"num_evict_kb":8828532,"num_promote":3783,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47836,"ondisk_log_size":47836,"up":32,"acting":32,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":32,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":37893248,"kb_used_data":37592872,"kb_used_omap":0,"kb_used_meta":278144,"kb_avail":528337792,"statfs":{"total":579820584960,"available":541017899008,"internally_reserved":0,"allocated":38495100928,"data_stored":38345262544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":284819456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1626810387,"num_objects":537,"num_object_clones":0,"num_object_copies":1074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":537,"num_whiteouts":0,"num_read":1173,"num_read_kb":102087,"num_write":2624,"num_write_kb":930886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":63,"num_flush_kb":174864,"num_evict":65,"num_evict_kb":181271,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":-2,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"24","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130732+0000","last_change":"2020-11-18T11:59:16.220885+0000","last_active":"2020-11-18T12:01:41.130732+0000","last_peered":"2020-11-18T12:01:41.130732+0000","last_clean":"2020-11-18T12:01:41.130732+0000","last_became_active":"2020-11-18T11:59:16.220460+0000","last_became_peered":"2020-11-18T11:59:16.220460+0000","last_unstale":"2020-11-18T12:01:41.130732+0000","last_undegraded":"2020-11-18T12:01:41.130732+0000","last_fullsized":"2020-11-18T12:01:41.130732+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"23","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130021+0000","last_change":"2020-11-18T11:59:16.499247+0000","last_active":"2020-11-18T12:01:41.130021+0000","last_peered":"2020-11-18T12:01:41.130021+0000","last_clean":"2020-11-18T12:01:41.130021+0000","last_became_active":"2020-11-18T11:59:16.498717+0000","last_became_peered":"2020-11-18T11:59:16.498717+0000","last_unstale":"2020-11-18T12:01:41.130021+0000","last_undegraded":"2020-11-18T12:01:41.130021+0000","last_fullsized":"2020-11-18T12:01:41.130021+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"34'5488","reported_seq":"7600","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:05.424086+0000","last_change":"2020-11-18T11:59:27.193656+0000","last_active":"2020-11-18T12:02:05.424086+0000","last_peered":"2020-11-18T12:02:05.424086+0000","last_clean":"2020-11-18T12:02:05.424086+0000","last_became_active":"2020-11-18T11:59:27.193348+0000","last_became_peered":"2020-11-18T11:59:27.193348+0000","last_unstale":"2020-11-18T12:02:05.424086+0000","last_undegraded":"2020-11-18T12:02:05.424086+0000","last_fullsized":"2020-11-18T12:02:05.424086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5488,"ondisk_log_size":5488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":133229742,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":1879,"num_read_kb":944,"num_write":4937,"num_write_kb":1137382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":2934,"num_flush":871,"num_flush_kb":2574747,"num_evict":807,"num_evict_kb":2383573,"num_promote":973,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"34'6592","reported_seq":"8537","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.465679+0000","last_change":"2020-11-18T11:59:25.400522+0000","last_active":"2020-11-18T12:02:15.465679+0000","last_peered":"2020-11-18T12:02:15.465679+0000","last_clean":"2020-11-18T12:02:15.465679+0000","last_became_active":"2020-11-18T11:59:25.399994+0000","last_became_peered":"2020-11-18T11:59:25.399994+0000","last_unstale":"2020-11-18T12:02:15.465679+0000","last_undegraded":"2020-11-18T12:02:15.465679+0000","last_fullsized":"2020-11-18T12:02:15.465679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6592,"ondisk_log_size":6592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7292778019,"num_objects":2447,"num_object_clones":0,"num_object_copies":4894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2447,"num_whiteouts":0,"num_read":3524,"num_read_kb":41917,"num_write":8202,"num_write_kb":4621789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"24","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130666+0000","last_change":"2020-11-18T11:59:16.498778+0000","last_active":"2020-11-18T12:01:41.130666+0000","last_peered":"2020-11-18T12:01:41.130666+0000","last_clean":"2020-11-18T12:01:41.130666+0000","last_became_active":"2020-11-18T11:59:16.498407+0000","last_became_peered":"2020-11-18T11:59:16.498407+0000","last_unstale":"2020-11-18T12:01:41.130666+0000","last_undegraded":"2020-11-18T12:01:41.130666+0000","last_fullsized":"2020-11-18T12:01:41.130666+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"34'5230","reported_seq":"7236","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:06.598034+0000","last_change":"2020-11-18T11:59:27.469664+0000","last_active":"2020-11-18T12:02:06.598034+0000","last_peered":"2020-11-18T12:02:06.598034+0000","last_clean":"2020-11-18T12:02:06.598034+0000","last_became_active":"2020-11-18T11:59:27.469083+0000","last_became_peered":"2020-11-18T11:59:27.469083+0000","last_unstale":"2020-11-18T12:02:06.598034+0000","last_undegraded":"2020-11-18T12:02:06.598034+0000","last_fullsized":"2020-11-18T12:02:06.598034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5230,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":139283403,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":1837,"num_read_kb":905,"num_write":4791,"num_write_kb":1124747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":3402,"num_flush":843,"num_flush_kb":2441950,"num_evict":772,"num_evict_kb":2231779,"num_promote":919,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"34'6685","reported_seq":"8719","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.457184+0000","last_change":"2020-11-18T11:59:25.186658+0000","last_active":"2020-11-18T12:02:15.457184+0000","last_peered":"2020-11-18T12:02:15.457184+0000","last_clean":"2020-11-18T12:02:15.457184+0000","last_became_active":"2020-11-18T11:59:25.186414+0000","last_became_peered":"2020-11-18T11:59:25.186414+0000","last_unstale":"2020-11-18T12:02:15.457184+0000","last_undegraded":"2020-11-18T12:02:15.457184+0000","last_fullsized":"2020-11-18T12:02:15.457184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6685,"ondisk_log_size":6685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7453896300,"num_objects":2479,"num_object_clones":0,"num_object_copies":4958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2479,"num_whiteouts":0,"num_read":3574,"num_read_kb":19412,"num_write":8297,"num_write_kb":4811913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"23","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130006+0000","last_change":"2020-11-18T11:59:16.499200+0000","last_active":"2020-11-18T12:01:41.130006+0000","last_peered":"2020-11-18T12:01:41.130006+0000","last_clean":"2020-11-18T12:01:41.130006+0000","last_became_active":"2020-11-18T11:59:16.498419+0000","last_became_peered":"2020-11-18T11:59:16.498419+0000","last_unstale":"2020-11-18T12:01:41.130006+0000","last_undegraded":"2020-11-18T12:01:41.130006+0000","last_fullsized":"2020-11-18T12:01:41.130006+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"34'5183","reported_seq":"7371","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:06.410297+0000","last_change":"2020-11-18T11:59:27.469738+0000","last_active":"2020-11-18T12:02:06.410297+0000","last_peered":"2020-11-18T12:02:06.410297+0000","last_clean":"2020-11-18T12:02:06.410297+0000","last_became_active":"2020-11-18T11:59:27.469241+0000","last_became_peered":"2020-11-18T11:59:27.469241+0000","last_unstale":"2020-11-18T12:02:06.410297+0000","last_undegraded":"2020-11-18T12:02:06.410297+0000","last_fullsized":"2020-11-18T12:02:06.410297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5183,"ondisk_log_size":5183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":150727528,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":1927,"num_read_kb":886,"num_write":4708,"num_write_kb":1109010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":2455,"num_flush":821,"num_flush_kb":2396213,"num_evict":630,"num_evict_kb":1841924,"num_promote":896,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"34'6480","reported_seq":"8496","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.440474+0000","last_change":"2020-11-18T11:59:25.400594+0000","last_active":"2020-11-18T12:02:15.440474+0000","last_peered":"2020-11-18T12:02:15.440474+0000","last_clean":"2020-11-18T12:02:15.440474+0000","last_became_active":"2020-11-18T11:59:25.400132+0000","last_became_peered":"2020-11-18T11:59:25.400132+0000","last_unstale":"2020-11-18T12:02:15.440474+0000","last_undegraded":"2020-11-18T12:02:15.440474+0000","last_fullsized":"2020-11-18T12:02:15.440474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6480,"ondisk_log_size":6480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7283473264,"num_objects":2438,"num_object_clones":0,"num_object_copies":4876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":3498,"num_read_kb":27406,"num_write":8030,"num_write_kb":4749930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.129702+0000","last_change":"2020-11-18T11:59:16.499029+0000","last_active":"2020-11-18T12:01:41.129702+0000","last_peered":"2020-11-18T12:01:41.129702+0000","last_clean":"2020-11-18T12:01:41.129702+0000","last_became_active":"2020-11-18T11:59:16.498581+0000","last_became_peered":"2020-11-18T11:59:16.498581+0000","last_unstale":"2020-11-18T12:01:41.129702+0000","last_undegraded":"2020-11-18T12:01:41.129702+0000","last_fullsized":"2020-11-18T12:01:41.129702+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"34'5553","reported_seq":"7740","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:07.814557+0000","last_change":"2020-11-18T11:59:27.192743+0000","last_active":"2020-11-18T12:02:07.814557+0000","last_peered":"2020-11-18T12:02:07.814557+0000","last_clean":"2020-11-18T12:02:07.814557+0000","last_became_active":"2020-11-18T11:59:27.192298+0000","last_became_peered":"2020-11-18T11:59:27.192298+0000","last_unstale":"2020-11-18T12:02:07.814557+0000","last_undegraded":"2020-11-18T12:02:07.814557+0000","last_fullsized":"2020-11-18T12:02:07.814557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5553,"ondisk_log_size":5553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":139702080,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":1945,"num_read_kb":972,"num_write":5029,"num_write_kb":1151459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":3818,"num_flush":891,"num_flush_kb":2600914,"num_evict":808,"num_evict_kb":2371256,"num_promote":995,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"34'6625","reported_seq":"8418","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.441145+0000","last_change":"2020-11-18T11:59:25.400403+0000","last_active":"2020-11-18T12:02:15.441145+0000","last_peered":"2020-11-18T12:02:15.441145+0000","last_clean":"2020-11-18T12:02:15.441145+0000","last_became_active":"2020-11-18T11:59:25.400017+0000","last_became_peered":"2020-11-18T11:59:25.400017+0000","last_unstale":"2020-11-18T12:02:15.441145+0000","last_undegraded":"2020-11-18T12:02:15.441145+0000","last_fullsized":"2020-11-18T12:02:15.441145+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6625,"ondisk_log_size":6625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7313935544,"num_objects":2442,"num_object_clones":0,"num_object_copies":4884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2442,"num_whiteouts":0,"num_read":3376,"num_read_kb":19124,"num_write":8248,"num_write_kb":4597811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.132577+0000","last_change":"2020-11-18T11:59:16.174660+0000","last_active":"2020-11-18T12:01:41.132577+0000","last_peered":"2020-11-18T12:01:41.132577+0000","last_clean":"2020-11-18T12:01:41.132577+0000","last_became_active":"2020-11-18T11:59:16.174201+0000","last_became_peered":"2020-11-18T11:59:16.174201+0000","last_unstale":"2020-11-18T12:01:41.132577+0000","last_undegraded":"2020-11-18T12:01:41.132577+0000","last_fullsized":"2020-11-18T12:01:41.132577+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.132535+0000","last_change":"2020-11-18T11:59:16.174652+0000","last_active":"2020-11-18T12:01:41.132535+0000","last_peered":"2020-11-18T12:01:41.132535+0000","last_clean":"2020-11-18T12:01:41.132535+0000","last_became_active":"2020-11-18T11:59:16.174176+0000","last_became_peered":"2020-11-18T11:59:16.174176+0000","last_unstale":"2020-11-18T12:01:41.132535+0000","last_undegraded":"2020-11-18T12:01:41.132535+0000","last_fullsized":"2020-11-18T12:01:41.132535+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.129724+0000","last_change":"2020-11-18T11:59:16.220833+0000","last_active":"2020-11-18T12:01:41.129724+0000","last_peered":"2020-11-18T12:01:41.129724+0000","last_clean":"2020-11-18T12:01:41.129724+0000","last_became_active":"2020-11-18T11:59:16.220457+0000","last_became_peered":"2020-11-18T11:59:16.220457+0000","last_unstale":"2020-11-18T12:01:41.129724+0000","last_undegraded":"2020-11-18T12:01:41.129724+0000","last_fullsized":"2020-11-18T12:01:41.129724+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":562942753,"num_objects":215,"num_object_clones":0,"num_object_copies":430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":190,"num_whiteouts":0,"num_read":7588,"num_read_kb":3707,"num_write":19465,"num_write_kb":4522598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":25,"num_bytes_hit_set_archive":12609,"num_flush":3426,"num_flush_kb":10013824,"num_evict":3017,"num_evict_kb":8828532,"num_promote":3783,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":509820928,"data_stored":505904942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21454,"ondisk_log_size":21454,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":4,"stat_sum":{"num_bytes":29344083127,"num_objects":9806,"num_object_clones":0,"num_object_copies":19612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9806,"num_whiteouts":0,"num_read":13972,"num_read_kb":107859,"num_write":32777,"num_write_kb":18781443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":37982961664,"data_stored":37838060392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26382,"ondisk_log_size":26382,"up":8,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640295,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9432696,"kb_used_data":9372000,"kb_used_omap":0,"kb_used_meta":51776,"kb_avail":84939144,"statfs":{"total":96636764160,"available":86977683456,"internally_reserved":0,"allocated":9596928000,"data_stored":9559886176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53018624},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.66200000000000003,"15min":0.64900000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60399999999999998,"15min":0.59399999999999997},"min":{"1min":0.41999999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82499999999999996,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.69699999999999995}]},{"osd":1,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67600000000000005,"15min":0.68200000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.77000000000000002,"15min":0.78300000000000003},"min":{"1min":0.51200000000000001,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.202,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.872}]},{"osd":2,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.80600000000000005,"15min":0.79000000000000004},"min":{"1min":0.52000000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.7010000000000001,"5min":1.7010000000000001,"15min":1.7010000000000001},"last":1.1679999999999999},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.86499999999999999,"15min":0.876},"min":{"1min":0.57099999999999995,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":1.026}]},{"osd":3,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.73499999999999999,"15min":0.73299999999999998},"min":{"1min":0.45100000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.80500000000000005,"15min":0.80900000000000005},"min":{"1min":0.56999999999999995,"5min":0.439,"15min":0.439},"max":{"1min":0.98399999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.98399999999999999}]},{"osd":4,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72299999999999998,"15min":0.74299999999999999},"min":{"1min":0.44800000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.875,"15min":0.876},"min":{"1min":0.64400000000000002,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.504,"5min":1.504,"15min":1.504},"last":1.1950000000000001}]}]},{"osd":4,"up_from":12,"seq":51539607592,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155880,"kb_used_data":119456,"kb_used_omap":0,"kb_used_meta":36416,"kb_avail":94215960,"statfs":{"total":96636764160,"available":96477143040,"internally_reserved":0,"allocated":122322944,"data_stored":121186256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37289984},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.68500000000000005,"15min":0.70599999999999996},"min":{"1min":0.40000000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.81499999999999995,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.64100000000000001,"15min":0.65200000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.878,"5min":1.131,"15min":1.131},"last":0.65300000000000002}]},{"osd":1,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.70099999999999996,"15min":0.70999999999999996},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.97499999999999998,"5min":1.145,"15min":1.145},"last":0.746},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.75700000000000001,"15min":0.77300000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.95599999999999996,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.68700000000000006}]},{"osd":2,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.81999999999999995,"15min":0.83799999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.02,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.84499999999999997,"15min":0.86099999999999999},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.054,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.879}]},{"osd":3,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70099999999999996,"15min":0.70999999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.78500000000000003,"15min":0.78900000000000003},"min":{"1min":0.57999999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.93600000000000005,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.94999999999999996}]},{"osd":5,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72799999999999998,"15min":0.72899999999999998},"min":{"1min":0.57599999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.873,"15min":0.88300000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.0349999999999999,"5min":1.264,"15min":1.264},"last":1.0209999999999999}]}]},{"osd":3,"up_from":12,"seq":51539607591,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4595176,"kb_used_data":4564824,"kb_used_omap":0,"kb_used_meta":28864,"kb_avail":89776664,"statfs":{"total":96636764160,"available":91931303936,"internally_reserved":0,"allocated":4674379776,"data_stored":4655592721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29556736},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.59699999999999998,"15min":0.58699999999999997},"min":{"1min":0.38400000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.421,"5min":1.421,"15min":1.421},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63,"15min":0.63500000000000001},"min":{"1min":0.45000000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91700000000000004,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.89200000000000002}]},{"osd":1,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.68700000000000006,"15min":0.68400000000000005},"min":{"1min":0.496,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.51700000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.7030000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":0.85599999999999998}]},{"osd":2,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.503,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.81399999999999995,"15min":0.83499999999999996},"min":{"1min":0.56599999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":1.0209999999999999}]},{"osd":4,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.46100000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.747},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.78000000000000003,"15min":0.79800000000000004},"min":{"1min":0.49099999999999999,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.046,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.93100000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.82899999999999996,"15min":0.80300000000000005},"min":{"1min":0.56399999999999995,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.137},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.83899999999999997,"15min":0.81499999999999995},"min":{"1min":0.65400000000000003,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.3199999999999998,"5min":2.3199999999999998,"15min":2.3199999999999998},"last":0.997}]}]},{"osd":2,"up_from":12,"seq":51539607591,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9531084,"kb_used_data":9470512,"kb_used_omap":0,"kb_used_meta":51328,"kb_avail":84840756,"statfs":{"total":96636764160,"available":86876934144,"internally_reserved":0,"allocated":9697804288,"data_stored":9661420712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52559872},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.502,"15min":0.5},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.438},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.62,"15min":0.622},"min":{"1min":0.40899999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93799999999999994,"5min":1.464,"15min":1.464},"last":0.48799999999999999}]},{"osd":1,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.68999999999999995,"15min":0.68899999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56799999999999995,"15min":0.57399999999999995},"min":{"1min":0.44500000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.67800000000000005,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.53300000000000003}]},{"osd":3,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.69299999999999995,"15min":0.69799999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0069999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.77000000000000002,"15min":0.78000000000000003},"min":{"1min":0.54300000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.026,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.79200000000000004}]},{"osd":4,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.89400000000000002,"15min":0.89800000000000002},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.101,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.876,"15min":0.88600000000000001},"min":{"1min":0.58899999999999997,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.294,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.75700000000000001}]},{"osd":5,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":2.2210000000000001,"15min":2.4700000000000002},"min":{"1min":0.60699999999999998,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.212,"5min":34.686999999999998,"15min":34.686999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.79800000000000004,"5min":2.266,"15min":2.5110000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.0029999999999999,"5min":34.798000000000002,"15min":34.798000000000002},"last":0.71999999999999997}]}]},{"osd":0,"up_from":12,"seq":51539607591,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155880,"kb_used_data":119456,"kb_used_omap":0,"kb_used_meta":36416,"kb_avail":94215960,"statfs":{"total":96636764160,"available":96477143040,"internally_reserved":0,"allocated":122322944,"data_stored":121186256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37289984},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.64000000000000001,"15min":0.65900000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0529999999999999,"5min":1.754,"15min":1.754},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.57999999999999996,"15min":0.56799999999999995},"min":{"1min":0.41999999999999998,"5min":0.37,"15min":0.37},"max":{"1min":1.639,"5min":1.639,"15min":1.639},"last":0.64100000000000001}]},{"osd":2,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.71499999999999997,"15min":0.72199999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.995,"5min":2.4079999999999999,"15min":2.4079999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.65200000000000002,"15min":0.64700000000000002},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.504}]},{"osd":3,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.76600000000000001,"15min":0.77400000000000002},"min":{"1min":0.51800000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1899999999999999,"5min":1.218,"15min":1.218},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.77500000000000002,"15min":0.79200000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.93600000000000005,"5min":1.181,"15min":1.181},"last":0.60099999999999998}]},{"osd":4,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.89400000000000002,"15min":0.91000000000000003},"min":{"1min":0.56799999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.069,"5min":1.9790000000000001,"15min":1.9790000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.91500000000000004,"15min":0.92700000000000005},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.3200000000000001,"5min":2.141,"15min":2.141},"last":0.92700000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.88700000000000001,"15min":0.88800000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.98699999999999999,"15min":1.002},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.276,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":0.79900000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640296,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14022532,"kb_used_data":13946624,"kb_used_omap":0,"kb_used_meta":73344,"kb_avail":80349308,"statfs":{"total":96636764160,"available":82277691392,"internally_reserved":0,"allocated":14281342976,"data_stored":14225990423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":75104256},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.58699999999999997,"15min":0.58099999999999996},"min":{"1min":0.40799999999999997,"5min":0.373,"15min":0.373},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.56899999999999995,"15min":0.55900000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.68400000000000005}]},{"osd":2,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57799999999999996,"15min":0.57699999999999996},"min":{"1min":0.41599999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.746,"5min":0.70199999999999996,"15min":0.69499999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.115,"5min":1.179,"15min":1.179},"last":0.61499999999999999}]},{"osd":3,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.753,"15min":0.76200000000000001},"min":{"1min":0.46300000000000002,"5min":0.441,"15min":0.441},"max":{"1min":1.1040000000000001,"5min":1.139,"15min":1.139},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.80000000000000004,"15min":0.80200000000000005},"min":{"1min":0.48099999999999998,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.96999999999999997}]},{"osd":4,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.90500000000000003,"15min":0.91700000000000004},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.0700000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":1.117},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.88900000000000001,"15min":0.89800000000000002},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.244,"5min":1.536,"15min":1.536},"last":0.93200000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:02:13 2020","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.93700000000000006,"15min":0.93200000000000005},"min":{"1min":0.70199999999999996,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.387,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.0249999999999999,"5min":0.95299999999999996,"15min":0.94099999999999995},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.48,"5min":1.48,"15min":1.48},"last":1.077}]}]}]}} , 2020-11-18T12:02:21.319 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-11-18T11:59:15.163082+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=11, tm_min=59, tm_sec=15, tm_wday=2, tm_yday=323, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=12, tm_min=2, tm_sec=16, tm_wday=2, tm_yday=323, tm_isdst=0) , 2020-11-18T12:02:21.319 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-11-18T11:59:15.163082+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=11, tm_min=59, tm_sec=15, tm_wday=2, tm_yday=323, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=12, tm_min=2, tm_sec=16, tm_wday=2, tm_yday=323, tm_isdst=0) 2020-11-18T18:42:47.183 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-18T12:03:26.330 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi045.front.sepia.ceph.com , 2020-11-18T12:03:26.330 INFO:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 , 2020-11-18T12:03:26.411 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi072.front.sepia.ceph.com 2020-11-18T18:42:47.183 WARNING:scrape:Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' , 2020-11-18T12:05:24.383 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2020-11-18T18:42:47.184 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status 2020-11-18T18:42:47.291 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-7273-gfbf83bb3 (fbf83bb377e36caf429a163721492f1b1c08deb9) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f14e3a605d4] , 2: (DeleteOp::_begin()+0x519) [0x55ce262f4099] 2020-11-18T18:42:47.292 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-18T12:02:21.315 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2020-11-18T12:02:20.101473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29907025880,"num_objects":10021,"num_object_clones":0,"num_object_copies":20042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9996,"num_whiteouts":0,"num_read":21560,"num_read_kb":111566,"num_write":52242,"num_write_kb":23304041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":25,"num_bytes_hit_set_archive":12609,"num_flush":3426,"num_flush_kb":10013824,"num_evict":3017,"num_evict_kb":8828532,"num_promote":3783,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47836,"ondisk_log_size":47836,"up":32,"acting":32,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":32,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":37893248,"kb_used_data":37592872,"kb_used_omap":0,"kb_used_meta":278144,"kb_avail":528337792,"statfs":{"total":579820584960,"available":541017899008,"internally_reserved":0,"allocated":38495100928,"data_stored":38345262544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":284819456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1626810387,"num_objects":537,"num_object_clones":0,"num_object_copies":1074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":537,"num_whiteouts":0,"num_read":1173,"num_read_kb":102087,"num_write":2624,"num_write_kb":930886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":63,"num_flush_kb":174864,"num_evict":65,"num_evict_kb":181271,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":-2,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"24","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130732+0000","last_change":"2020-11-18T11:59:16.220885+0000","last_active":"2020-11-18T12:01:41.130732+0000","last_peered":"2020-11-18T12:01:41.130732+0000","last_clean":"2020-11-18T12:01:41.130732+0000","last_became_active":"2020-11-18T11:59:16.220460+0000","last_became_peered":"2020-11-18T11:59:16.220460+0000","last_unstale":"2020-11-18T12:01:41.130732+0000","last_undegraded":"2020-11-18T12:01:41.130732+0000","last_fullsized":"2020-11-18T12:01:41.130732+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"23","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130021+0000","last_change":"2020-11-18T11:59:16.499247+0000","last_active":"2020-11-18T12:01:41.130021+0000","last_peered":"2020-11-18T12:01:41.130021+0000","last_clean":"2020-11-18T12:01:41.130021+0000","last_became_active":"2020-11-18T11:59:16.498717+0000","last_became_peered":"2020-11-18T11:59:16.498717+0000","last_unstale":"2020-11-18T12:01:41.130021+0000","last_undegraded":"2020-11-18T12:01:41.130021+0000","last_fullsized":"2020-11-18T12:01:41.130021+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"34'5488","reported_seq":"7600","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:05.424086+0000","last_change":"2020-11-18T11:59:27.193656+0000","last_active":"2020-11-18T12:02:05.424086+0000","last_peered":"2020-11-18T12:02:05.424086+0000","last_clean":"2020-11-18T12:02:05.424086+0000","last_became_active":"2020-11-18T11:59:27.193348+0000","last_became_peered":"2020-11-18T11:59:27.193348+0000","last_unstale":"2020-11-18T12:02:05.424086+0000","last_undegraded":"2020-11-18T12:02:05.424086+0000","last_fullsized":"2020-11-18T12:02:05.424086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5488,"ondisk_log_size":5488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":133229742,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":1879,"num_read_kb":944,"num_write":4937,"num_write_kb":1137382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":2934,"num_flush":871,"num_flush_kb":2574747,"num_evict":807,"num_evict_kb":2383573,"num_promote":973,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"34'6592","reported_seq":"8537","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.465679+0000","last_change":"2020-11-18T11:59:25.400522+0000","last_active":"2020-11-18T12:02:15.465679+0000","last_peered":"2020-11-18T12:02:15.465679+0000","last_clean":"2020-11-18T12:02:15.465679+0000","last_became_active":"2020-11-18T11:59:25.399994+0000","last_became_peered":"2020-11-18T11:59:25.399994+0000","last_unstale":"2020-11-18T12:02:15.465679+0000","last_undegraded":"2020-11-18T12:02:15.465679+0000","last_fullsized":"2020-11-18T12:02:15.465679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6592,"ondisk_log_size":6592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7292778019,"num_objects":2447,"num_object_clones":0,"num_object_copies":4894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2447,"num_whiteouts":0,"num_read":3524,"num_read_kb":41917,"num_write":8202,"num_write_kb":4621789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"24","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130666+0000","last_change":"2020-11-18T11:59:16.498778+0000","last_active":"2020-11-18T12:01:41.130666+0000","last_peered":"2020-11-18T12:01:41.130666+0000","last_clean":"2020-11-18T12:01:41.130666+0000","last_became_active":"2020-11-18T11:59:16.498407+0000","last_became_peered":"2020-11-18T11:59:16.498407+0000","last_unstale":"2020-11-18T12:01:41.130666+0000","last_undegraded":"2020-11-18T12:01:41.130666+0000","last_fullsized":"2020-11-18T12:01:41.130666+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"34'5230","reported_seq":"7236","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:06.598034+0000","last_change":"2020-11-18T11:59:27.469664+0000","last_active":"2020-11-18T12:02:06.598034+0000","last_peered":"2020-11-18T12:02:06.598034+0000","last_clean":"2020-11-18T12:02:06.598034+0000","last_became_active":"2020-11-18T11:59:27.469083+0000","last_became_peered":"2020-11-18T11:59:27.469083+0000","last_unstale":"2020-11-18T12:02:06.598034+0000","last_undegraded":"2020-11-18T12:02:06.598034+0000","last_fullsized":"2020-11-18T12:02:06.598034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5230,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":139283403,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":1837,"num_read_kb":905,"num_write":4791,"num_write_kb":1124747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":3402,"num_flush":843,"num_flush_kb":2441950,"num_evict":772,"num_evict_kb":2231779,"num_promote":919,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"34'6685","reported_seq":"8719","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.457184+0000","last_change":"2020-11-18T11:59:25.186658+0000","last_active":"2020-11-18T12:02:15.457184+0000","last_peered":"2020-11-18T12:02:15.457184+0000","last_clean":"2020-11-18T12:02:15.457184+0000","last_became_active":"2020-11-18T11:59:25.186414+0000","last_became_peered":"2020-11-18T11:59:25.186414+0000","last_unstale":"2020-11-18T12:02:15.457184+0000","last_undegraded":"2020-11-18T12:02:15.457184+0000","last_fullsized":"2020-11-18T12:02:15.457184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6685,"ondisk_log_size":6685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7453896300,"num_objects":2479,"num_object_clones":0,"num_object_copies":4958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2479,"num_whiteouts":0,"num_read":3574,"num_read_kb":19412,"num_write":8297,"num_write_kb":4811913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"23","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130006+0000","last_change":"2020-11-18T11:59:16.499200+0000","last_active":"2020-11-18T12:01:41.130006+0000","last_peered":"2020-11-18T12:01:41.130006+0000","last_clean":"2020-11-18T12:01:41.130006+0000","last_became_active":"2020-11-18T11:59:16.498419+0000","last_became_peered":"2020-11-18T11:59:16.498419+0000","last_unstale":"2020-11-18T12:01:41.130006+0000","last_undegraded":"2020-11-18T12:01:41.130006+0000","last_fullsized":"2020-11-18T12:01:41.130006+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"34'5183","reported_seq":"7371","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:06.410297+0000","last_change":"2020-11-18T11:59:27.469738+0000","last_active":"2020-11-18T12:02:06.410297+0000","last_peered":"2020-11-18T12:02:06.410297+0000","last_clean":"2020-11-18T12:02:06.410297+0000","last_became_active":"2020-11-18T11:59:27.469241+0000","last_became_peered":"2020-11-18T11:59:27.469241+0000","last_unstale":"2020-11-18T12:02:06.410297+0000","last_undegraded":"2020-11-18T12:02:06.410297+0000","last_fullsized":"2020-11-18T12:02:06.410297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5183,"ondisk_log_size":5183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":150727528,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":1927,"num_read_kb":886,"num_write":4708,"num_write_kb":1109010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":2455,"num_flush":821,"num_flush_kb":2396213,"num_evict":630,"num_evict_kb":1841924,"num_promote":896,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"34'6480","reported_seq":"8496","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.440474+0000","last_change":"2020-11-18T11:59:25.400594+0000","last_active":"2020-11-18T12:02:15.440474+0000","last_peered":"2020-11-18T12:02:15.440474+0000","last_clean":"2020-11-18T12:02:15.440474+0000","last_became_active":"2020-11-18T11:59:25.400132+0000","last_became_peered":"2020-11-18T11:59:25.400132+0000","last_unstale":"2020-11-18T12:02:15.440474+0000","last_undegraded":"2020-11-18T12:02:15.440474+0000","last_fullsized":"2020-11-18T12:02:15.440474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6480,"ondisk_log_size":6480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7283473264,"num_objects":2438,"num_object_clones":0,"num_object_copies":4876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":3498,"num_read_kb":27406,"num_write":8030,"num_write_kb":4749930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.129702+0000","last_change":"2020-11-18T11:59:16.499029+0000","last_active":"2020-11-18T12:01:41.129702+0000","last_peered":"2020-11-18T12:01:41.129702+0000","last_clean":"2020-11-18T12:01:41.129702+0000","last_became_active":"2020-11-18T11:59:16.498581+0000","last_became_peered":"2020-11-18T11:59:16.498581+0000","last_unstale":"2020-11-18T12:01:41.129702+0000","last_undegraded":"2020-11-18T12:01:41.129702+0000","last_fullsized":"2020-11-18T12:01:41.129702+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"34'5553","reported_seq":"7740","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:07.814557+0000","last_change":"2020-11-18T11:59:27.192743+0000","last_active":"2020-11-18T12:02:07.814557+0000","last_peered":"2020-11-18T12:02:07.814557+0000","last_clean":"2020-11-18T12:02:07.814557+0000","last_became_active":"2020-11-18T11:59:27.192298+0000","last_became_peered":"2020-11-18T11:59:27.192298+0000","last_unstale":"2020-11-18T12:02:07.814557+0000","last_undegraded":"2020-11-18T12:02:07.814557+0000","last_fullsized":"2020-11-18T12:02:07.814557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5553,"ondisk_log_size":5553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":139702080,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":1945,"num_read_kb":972,"num_write":5029,"num_write_kb":1151459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":3818,"num_flush":891,"num_flush_kb":2600914,"num_evict":808,"num_evict_kb":2371256,"num_promote":995,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"34'6625","reported_seq":"8418","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.441145+0000","last_change":"2020-11-18T11:59:25.400403+0000","last_active":"2020-11-18T12:02:15.441145+0000","last_peered":"2020-11-18T12:02:15.441145+0000","last_clean":"2020-11-18T12:02:15.441145+0000","last_became_active":"2020-11-18T11:59:25.400017+0000","last_became_peered":"2020-11-18T11:59:25.400017+0000","last_unstale":"2020-11-18T12:02:15.441145+0000","last_undegraded":"2020-11-18T12:02:15.441145+0000","last_fullsized":"2020-11-18T12:02:15.441145+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6625,"ondisk_log_size":6625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7313935544,"num_objects":2442,"num_object_clones":0,"num_object_copies":4884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2442,"num_whiteouts":0,"num_read":3376,"num_read_kb":19124,"num_write":8248,"num_write_kb":4597811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.132577+0000","last_change":"2020-11-18T11:59:16.174660+0000","last_active":"2020-11-18T12:01:41.132577+0000","last_peered":"2020-11-18T12:01:41.132577+0000","last_clean":"2020-11-18T12:01:41.132577+0000","last_became_active":"2020-11-18T11:59:16.174201+0000","last_became_peered":"2020-11-18T11:59:16.174201+0000","last_unstale":"2020-11-18T12:01:41.132577+0000","last_undegraded":"2020-11-18T12:01:41.132577+0000","last_fullsized":"2020-11-18T12:01:41.132577+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.132535+0000","last_change":"2020-11-18T11:59:16.174652+0000","last_active":"2020-11-18T12:01:41.132535+0000","last_peered":"2020-11-18T12:01:41.132535+0000","last_clean":"2020-11-18T12:01:41.132535+0000","last_became_active":"2020-11-18T11:59:16.174176+0000","last_became_peered":"2020-11-18T11:59:16.174176+0000","last_unstale":"2020-11-18T12:01:41.132535+0000","last_undegraded":"2020-11-18T12:01:41.132535+0000","last_fullsized":"2020-11-18T12:01:41.132535+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.129724+0000","last_change":"2020-11-18T11:59:16.220833+0000","last_active":"2020-11-18T12:01:41.129724+0000","last_peered":"2020-11-18T12:01:41.129724+0000","last_clean":"2020-11-18T12:01:41.129724+0000","last_became_active":"2020-11-18T11:59:16.220457+0000","last_became_peered":"2020-11-18T11:59:16.220457+0000","last_unstale":"2020-11-18T12:01:41.129724+0000","last_undegraded":"2020-11-18T12:01:41.129724+0000","last_fullsized":"2020-11-18T12:01:41.129724+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":562942753,"num_objects":215,"num_object_clones":0,"num_object_copies":430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":190,"num_whiteouts":0,"num_read":7588,"num_read_kb":3707,"num_write":19465,"num_write_kb":4522598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":25,"num_bytes_hit_set_archive":12609,"num_flush":3426,"num_flush_kb":10013824,"num_evict":3017,"num_evict_kb":8828532,"num_promote":3783,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":509820928,"data_stored":505904942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21454,"ondisk_log_size":21454,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":4,"stat_sum":{"num_bytes":29344083127,"num_objects":9806,"num_object_clones":0,"num_object_copies":19612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9806,"num_whiteouts":0,"num_read":13972,"num_read_kb":107859,"num_write":32777,"num_write_kb":18781443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":37982961664,"data_stored":37838060392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26382,"ondisk_log_size":26382,"up":8,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640295,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9432696,"kb_used_data":9372000,"kb_used_omap":0,"kb_used_meta":51776,"kb_avail":84939144,"statfs":{"total":96636764160,"available":86977683456,"internally_reserved":0,"allocated":9596928000,"data_stored":9559886176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53018624},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.66200000000000003,"15min":0.64900000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60399999999999998,"15min":0.59399999999999997},"min":{"1min":0.41999999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82499999999999996,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.69699999999999995}]},{"osd":1,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67600000000000005,"15min":0.68200000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.77000000000000002,"15min":0.78300000000000003},"min":{"1min":0.51200000000000001,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.202,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.872}]},{"osd":2,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.80600000000000005,"15min":0.79000000000000004},"min":{"1min":0.52000000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.7010000000000001,"5min":1.7010000000000001,"15min":1.7010000000000001},"last":1.1679999999999999},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.86499999999999999,"15min":0.876},"min":{"1min":0.57099999999999995,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":1.026}]},{"osd":3,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.73499999999999999,"15min":0.73299999999999998},"min":{"1min":0.45100000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.80500000000000005,"15min":0.80900000000000005},"min":{"1min":0.56999999999999995,"5min":0.439,"15min":0.439},"max":{"1min":0.98399999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.98399999999999999}]},{"osd":4,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72299999999999998,"15min":0.74299999999999999},"min":{"1min":0.44800000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.875,"15min":0.876},"min":{"1min":0.64400000000000002,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.504,"5min":1.504,"15min":1.504},"last":1.1950000000000001}]}]},{"osd":4,"up_from":12,"seq":51539607592,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155880,"kb_used_data":119456,"kb_used_omap":0,"kb_used_meta":36416,"kb_avail":94215960,"statfs":{"total":96636764160,"available":96477143040,"internally_reserved":0,"allocated":122322944,"data_stored":121186256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37289984},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.68500000000000005,"15min":0.70599999999999996},"min":{"1min":0.40000000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.81499999999999995,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.64100000000000001,"15min":0.65200000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.878,"5min":1.131,"15min":1.131},"last":0.65300000000000002}]},{"osd":1,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.70099999999999996,"15min":0.70999999999999996},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.97499999999999998,"5min":1.145,"15min":1.145},"last":0.746},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.75700000000000001,"15min":0.77300000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.95599999999999996,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.68700000000000006}]},{"osd":2,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.81999999999999995,"15min":0.83799999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.02,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.84499999999999997,"15min":0.86099999999999999},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.054,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.879}]},{"osd":3,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70099999999999996,"15min":0.70999999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.78500000000000003,"15min":0.78900000000000003},"min":{"1min":0.57999999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.93600000000000005,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.94999999999999996}]},{"osd":5,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72799999999999998,"15min":0.72899999999999998},"min":{"1min":0.57599999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.873,"15min":0.88300000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.0349999999999999,"5min":1.264,"15min":1.264},"last":1.0209999999999999}]}]},{"osd":3,"up_from":12,"seq":51539607591,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4595176,"kb_used_data":4564824,"kb_used_omap":0,"kb_used_meta":28864,"kb_avail":89776664,"statfs":{"total":96636764160,"available":91931303936,"internally_reserved":0,"allocated":4674379776,"data_stored":4655592721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29556736},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.59699999999999998,"15min":0.58699999999999997},"min":{"1min":0.38400000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.421,"5min":1.421,"15min":1.421},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63,"15min":0.63500000000000001},"min":{"1min":0.45000000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91700000000000004,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.89200000000000002}]},{"osd":1,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.68700000000000006,"15min":0.68400000000000005},"min":{"1min":0.496,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.51700000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.7030000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":0.85599999999999998}]},{"osd":2,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.503,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.81399999999999995,"15min":0.83499999999999996},"min":{"1min":0.56599999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":1.0209999999999999}]},{"osd":4,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.46100000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.747},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.78000000000000003,"15min":0.79800000000000004},"min":{"1min":0.49099999999999999,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.046,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.93100000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.82899999999999996,"15min":0.80300000000000005},"min":{"1min":0.56399999999999995,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.137},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.83899999999999997,"15min":0.81499999999999995},"min":{"1min":0.65400000000000003,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.3199999999999998,"5min":2.3199999999999998,"15min":2.3199999999999998},"last":0.997}]}]},{"osd":2,"up_from":12,"seq":51539607591,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9531084,"kb_used_data":9470512,"kb_used_omap":0,"kb_used_meta":51328,"kb_avail":84840756,"statfs":{"total":96636764160,"available":86876934144,"internally_reserved":0,"allocated":9697804288,"data_stored":9661420712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52559872},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.502,"15min":0.5},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.438},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.62,"15min":0.622},"min":{"1min":0.40899999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93799999999999994,"5min":1.464,"15min":1.464},"last":0.48799999999999999}]},{"osd":1,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.68999999999999995,"15min":0.68899999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56799999999999995,"15min":0.57399999999999995},"min":{"1min":0.44500000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.67800000000000005,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.53300000000000003}]},{"osd":3,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.69299999999999995,"15min":0.69799999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0069999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.77000000000000002,"15min":0.78000000000000003},"min":{"1min":0.54300000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.026,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.79200000000000004}]},{"osd":4,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.89400000000000002,"15min":0.89800000000000002},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.101,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.876,"15min":0.88600000000000001},"min":{"1min":0.58899999999999997,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.294,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.75700000000000001}]},{"osd":5,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":2.2210000000000001,"15min":2.4700000000000002},"min":{"1min":0.60699999999999998,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.212,"5min":34.686999999999998,"15min":34.686999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.79800000000000004,"5min":2.266,"15min":2.5110000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.0029999999999999,"5min":34.798000000000002,"15min":34.798000000000002},"last":0.71999999999999997}]}]},{"osd":0,"up_from":12,"seq":51539607591,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155880,"kb_used_data":119456,"kb_used_omap":0,"kb_used_meta":36416,"kb_avail":94215960,"statfs":{"total":96636764160,"available":96477143040,"internally_reserved":0,"allocated":122322944,"data_stored":121186256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37289984},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.64000000000000001,"15min":0.65900000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0529999999999999,"5min":1.754,"15min":1.754},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.57999999999999996,"15min":0.56799999999999995},"min":{"1min":0.41999999999999998,"5min":0.37,"15min":0.37},"max":{"1min":1.639,"5min":1.639,"15min":1.639},"last":0.64100000000000001}]},{"osd":2,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.71499999999999997,"15min":0.72199999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.995,"5min":2.4079999999999999,"15min":2.4079999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.65200000000000002,"15min":0.64700000000000002},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.504}]},{"osd":3,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.76600000000000001,"15min":0.77400000000000002},"min":{"1min":0.51800000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1899999999999999,"5min":1.218,"15min":1.218},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.77500000000000002,"15min":0.79200000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.93600000000000005,"5min":1.181,"15min":1.181},"last":0.60099999999999998}]},{"osd":4,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.89400000000000002,"15min":0.91000000000000003},"min":{"1min":0.56799999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.069,"5min":1.9790000000000001,"15min":1.9790000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.91500000000000004,"15min":0.92700000000000005},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.3200000000000001,"5min":2.141,"15min":2.141},"last":0.92700000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.88700000000000001,"15min":0.88800000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.98699999999999999,"15min":1.002},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.276,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":0.79900000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640296,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14022532,"kb_used_data":13946624,"kb_used_omap":0,"kb_used_meta":73344,"kb_avail":80349308,"statfs":{"total":96636764160,"available":82277691392,"internally_reserved":0,"allocated":14281342976,"data_stored":14225990423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":75104256},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.58699999999999997,"15min":0.58099999999999996},"min":{"1min":0.40799999999999997,"5min":0.373,"15min":0.373},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.56899999999999995,"15min":0.55900000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.68400000000000005}]},{"osd":2,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57799999999999996,"15min":0.57699999999999996},"min":{"1min":0.41599999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.746,"5min":0.70199999999999996,"15min":0.69499999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.115,"5min":1.179,"15min":1.179},"last":0.61499999999999999}]},{"osd":3,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.753,"15min":0.76200000000000001},"min":{"1min":0.46300000000000002,"5min":0.441,"15min":0.441},"max":{"1min":1.1040000000000001,"5min":1.139,"15min":1.139},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.80000000000000004,"15min":0.80200000000000005},"min":{"1min":0.48099999999999998,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.96999999999999997}]},{"osd":4,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.90500000000000003,"15min":0.91700000000000004},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.0700000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":1.117},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.88900000000000001,"15min":0.89800000000000002},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.244,"5min":1.536,"15min":1.536},"last":0.93200000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:02:13 2020","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.93700000000000006,"15min":0.93200000000000005},"min":{"1min":0.70199999999999996,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.387,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.0249999999999999,"5min":0.95299999999999996,"15min":0.94099999999999995},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.48,"5min":1.48,"15min":1.48},"last":1.077}]}]}]}} , 2020-11-18T12:02:21.319 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-11-18T11:59:15.163082+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=11, tm_min=59, tm_sec=15, tm_wday=2, tm_yday=323, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=12, tm_min=2, tm_sec=16, tm_wday=2, tm_yday=323, tm_isdst=0) , 2020-11-18T12:02:21.319 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-11-18T11:59:15.163082+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=11, tm_min=59, tm_sec=15, tm_wday=2, tm_yday=323, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=12, tm_min=2, tm_sec=16, tm_wday=2, tm_yday=323, tm_isdst=0) 2020-11-18T18:42:47.294 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-18T12:03:26.330 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi045.front.sepia.ceph.com , 2020-11-18T12:03:26.330 INFO:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 , 2020-11-18T12:03:26.411 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi072.front.sepia.ceph.com 2020-11-18T18:42:47.294 WARNING:scrape:Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' , 2020-11-18T12:05:24.383 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2020-11-18T18:42:47.295 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status 2020-11-18T18:42:47.399 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-7273-gfbf83bb3 (fbf83bb377e36caf429a163721492f1b1c08deb9) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f14e3a605d4] , 2: (DeleteOp::_begin()+0x519) [0x55ce262f4099] 2020-11-18T18:42:47.401 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-18T12:02:21.315 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2020-11-18T12:02:20.101473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29907025880,"num_objects":10021,"num_object_clones":0,"num_object_copies":20042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9996,"num_whiteouts":0,"num_read":21560,"num_read_kb":111566,"num_write":52242,"num_write_kb":23304041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":25,"num_bytes_hit_set_archive":12609,"num_flush":3426,"num_flush_kb":10013824,"num_evict":3017,"num_evict_kb":8828532,"num_promote":3783,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47836,"ondisk_log_size":47836,"up":32,"acting":32,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":32,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":37893248,"kb_used_data":37592872,"kb_used_omap":0,"kb_used_meta":278144,"kb_avail":528337792,"statfs":{"total":579820584960,"available":541017899008,"internally_reserved":0,"allocated":38495100928,"data_stored":38345262544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":284819456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1626810387,"num_objects":537,"num_object_clones":0,"num_object_copies":1074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":537,"num_whiteouts":0,"num_read":1173,"num_read_kb":102087,"num_write":2624,"num_write_kb":930886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":63,"num_flush_kb":174864,"num_evict":65,"num_evict_kb":181271,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":-2,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"24","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130732+0000","last_change":"2020-11-18T11:59:16.220885+0000","last_active":"2020-11-18T12:01:41.130732+0000","last_peered":"2020-11-18T12:01:41.130732+0000","last_clean":"2020-11-18T12:01:41.130732+0000","last_became_active":"2020-11-18T11:59:16.220460+0000","last_became_peered":"2020-11-18T11:59:16.220460+0000","last_unstale":"2020-11-18T12:01:41.130732+0000","last_undegraded":"2020-11-18T12:01:41.130732+0000","last_fullsized":"2020-11-18T12:01:41.130732+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"23","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130021+0000","last_change":"2020-11-18T11:59:16.499247+0000","last_active":"2020-11-18T12:01:41.130021+0000","last_peered":"2020-11-18T12:01:41.130021+0000","last_clean":"2020-11-18T12:01:41.130021+0000","last_became_active":"2020-11-18T11:59:16.498717+0000","last_became_peered":"2020-11-18T11:59:16.498717+0000","last_unstale":"2020-11-18T12:01:41.130021+0000","last_undegraded":"2020-11-18T12:01:41.130021+0000","last_fullsized":"2020-11-18T12:01:41.130021+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"34'5488","reported_seq":"7600","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:05.424086+0000","last_change":"2020-11-18T11:59:27.193656+0000","last_active":"2020-11-18T12:02:05.424086+0000","last_peered":"2020-11-18T12:02:05.424086+0000","last_clean":"2020-11-18T12:02:05.424086+0000","last_became_active":"2020-11-18T11:59:27.193348+0000","last_became_peered":"2020-11-18T11:59:27.193348+0000","last_unstale":"2020-11-18T12:02:05.424086+0000","last_undegraded":"2020-11-18T12:02:05.424086+0000","last_fullsized":"2020-11-18T12:02:05.424086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5488,"ondisk_log_size":5488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":133229742,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":1879,"num_read_kb":944,"num_write":4937,"num_write_kb":1137382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":2934,"num_flush":871,"num_flush_kb":2574747,"num_evict":807,"num_evict_kb":2383573,"num_promote":973,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"34'6592","reported_seq":"8537","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.465679+0000","last_change":"2020-11-18T11:59:25.400522+0000","last_active":"2020-11-18T12:02:15.465679+0000","last_peered":"2020-11-18T12:02:15.465679+0000","last_clean":"2020-11-18T12:02:15.465679+0000","last_became_active":"2020-11-18T11:59:25.399994+0000","last_became_peered":"2020-11-18T11:59:25.399994+0000","last_unstale":"2020-11-18T12:02:15.465679+0000","last_undegraded":"2020-11-18T12:02:15.465679+0000","last_fullsized":"2020-11-18T12:02:15.465679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6592,"ondisk_log_size":6592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7292778019,"num_objects":2447,"num_object_clones":0,"num_object_copies":4894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2447,"num_whiteouts":0,"num_read":3524,"num_read_kb":41917,"num_write":8202,"num_write_kb":4621789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"24","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130666+0000","last_change":"2020-11-18T11:59:16.498778+0000","last_active":"2020-11-18T12:01:41.130666+0000","last_peered":"2020-11-18T12:01:41.130666+0000","last_clean":"2020-11-18T12:01:41.130666+0000","last_became_active":"2020-11-18T11:59:16.498407+0000","last_became_peered":"2020-11-18T11:59:16.498407+0000","last_unstale":"2020-11-18T12:01:41.130666+0000","last_undegraded":"2020-11-18T12:01:41.130666+0000","last_fullsized":"2020-11-18T12:01:41.130666+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"34'5230","reported_seq":"7236","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:06.598034+0000","last_change":"2020-11-18T11:59:27.469664+0000","last_active":"2020-11-18T12:02:06.598034+0000","last_peered":"2020-11-18T12:02:06.598034+0000","last_clean":"2020-11-18T12:02:06.598034+0000","last_became_active":"2020-11-18T11:59:27.469083+0000","last_became_peered":"2020-11-18T11:59:27.469083+0000","last_unstale":"2020-11-18T12:02:06.598034+0000","last_undegraded":"2020-11-18T12:02:06.598034+0000","last_fullsized":"2020-11-18T12:02:06.598034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5230,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":139283403,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":1837,"num_read_kb":905,"num_write":4791,"num_write_kb":1124747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":3402,"num_flush":843,"num_flush_kb":2441950,"num_evict":772,"num_evict_kb":2231779,"num_promote":919,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"34'6685","reported_seq":"8719","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.457184+0000","last_change":"2020-11-18T11:59:25.186658+0000","last_active":"2020-11-18T12:02:15.457184+0000","last_peered":"2020-11-18T12:02:15.457184+0000","last_clean":"2020-11-18T12:02:15.457184+0000","last_became_active":"2020-11-18T11:59:25.186414+0000","last_became_peered":"2020-11-18T11:59:25.186414+0000","last_unstale":"2020-11-18T12:02:15.457184+0000","last_undegraded":"2020-11-18T12:02:15.457184+0000","last_fullsized":"2020-11-18T12:02:15.457184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6685,"ondisk_log_size":6685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7453896300,"num_objects":2479,"num_object_clones":0,"num_object_copies":4958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2479,"num_whiteouts":0,"num_read":3574,"num_read_kb":19412,"num_write":8297,"num_write_kb":4811913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"23","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130006+0000","last_change":"2020-11-18T11:59:16.499200+0000","last_active":"2020-11-18T12:01:41.130006+0000","last_peered":"2020-11-18T12:01:41.130006+0000","last_clean":"2020-11-18T12:01:41.130006+0000","last_became_active":"2020-11-18T11:59:16.498419+0000","last_became_peered":"2020-11-18T11:59:16.498419+0000","last_unstale":"2020-11-18T12:01:41.130006+0000","last_undegraded":"2020-11-18T12:01:41.130006+0000","last_fullsized":"2020-11-18T12:01:41.130006+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"34'5183","reported_seq":"7371","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:06.410297+0000","last_change":"2020-11-18T11:59:27.469738+0000","last_active":"2020-11-18T12:02:06.410297+0000","last_peered":"2020-11-18T12:02:06.410297+0000","last_clean":"2020-11-18T12:02:06.410297+0000","last_became_active":"2020-11-18T11:59:27.469241+0000","last_became_peered":"2020-11-18T11:59:27.469241+0000","last_unstale":"2020-11-18T12:02:06.410297+0000","last_undegraded":"2020-11-18T12:02:06.410297+0000","last_fullsized":"2020-11-18T12:02:06.410297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5183,"ondisk_log_size":5183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":150727528,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":1927,"num_read_kb":886,"num_write":4708,"num_write_kb":1109010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":2455,"num_flush":821,"num_flush_kb":2396213,"num_evict":630,"num_evict_kb":1841924,"num_promote":896,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"34'6480","reported_seq":"8496","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.440474+0000","last_change":"2020-11-18T11:59:25.400594+0000","last_active":"2020-11-18T12:02:15.440474+0000","last_peered":"2020-11-18T12:02:15.440474+0000","last_clean":"2020-11-18T12:02:15.440474+0000","last_became_active":"2020-11-18T11:59:25.400132+0000","last_became_peered":"2020-11-18T11:59:25.400132+0000","last_unstale":"2020-11-18T12:02:15.440474+0000","last_undegraded":"2020-11-18T12:02:15.440474+0000","last_fullsized":"2020-11-18T12:02:15.440474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6480,"ondisk_log_size":6480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7283473264,"num_objects":2438,"num_object_clones":0,"num_object_copies":4876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":3498,"num_read_kb":27406,"num_write":8030,"num_write_kb":4749930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.129702+0000","last_change":"2020-11-18T11:59:16.499029+0000","last_active":"2020-11-18T12:01:41.129702+0000","last_peered":"2020-11-18T12:01:41.129702+0000","last_clean":"2020-11-18T12:01:41.129702+0000","last_became_active":"2020-11-18T11:59:16.498581+0000","last_became_peered":"2020-11-18T11:59:16.498581+0000","last_unstale":"2020-11-18T12:01:41.129702+0000","last_undegraded":"2020-11-18T12:01:41.129702+0000","last_fullsized":"2020-11-18T12:01:41.129702+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"34'5553","reported_seq":"7740","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:07.814557+0000","last_change":"2020-11-18T11:59:27.192743+0000","last_active":"2020-11-18T12:02:07.814557+0000","last_peered":"2020-11-18T12:02:07.814557+0000","last_clean":"2020-11-18T12:02:07.814557+0000","last_became_active":"2020-11-18T11:59:27.192298+0000","last_became_peered":"2020-11-18T11:59:27.192298+0000","last_unstale":"2020-11-18T12:02:07.814557+0000","last_undegraded":"2020-11-18T12:02:07.814557+0000","last_fullsized":"2020-11-18T12:02:07.814557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5553,"ondisk_log_size":5553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":139702080,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":1945,"num_read_kb":972,"num_write":5029,"num_write_kb":1151459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":3818,"num_flush":891,"num_flush_kb":2600914,"num_evict":808,"num_evict_kb":2371256,"num_promote":995,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"34'6625","reported_seq":"8418","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.441145+0000","last_change":"2020-11-18T11:59:25.400403+0000","last_active":"2020-11-18T12:02:15.441145+0000","last_peered":"2020-11-18T12:02:15.441145+0000","last_clean":"2020-11-18T12:02:15.441145+0000","last_became_active":"2020-11-18T11:59:25.400017+0000","last_became_peered":"2020-11-18T11:59:25.400017+0000","last_unstale":"2020-11-18T12:02:15.441145+0000","last_undegraded":"2020-11-18T12:02:15.441145+0000","last_fullsized":"2020-11-18T12:02:15.441145+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6625,"ondisk_log_size":6625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7313935544,"num_objects":2442,"num_object_clones":0,"num_object_copies":4884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2442,"num_whiteouts":0,"num_read":3376,"num_read_kb":19124,"num_write":8248,"num_write_kb":4597811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.132577+0000","last_change":"2020-11-18T11:59:16.174660+0000","last_active":"2020-11-18T12:01:41.132577+0000","last_peered":"2020-11-18T12:01:41.132577+0000","last_clean":"2020-11-18T12:01:41.132577+0000","last_became_active":"2020-11-18T11:59:16.174201+0000","last_became_peered":"2020-11-18T11:59:16.174201+0000","last_unstale":"2020-11-18T12:01:41.132577+0000","last_undegraded":"2020-11-18T12:01:41.132577+0000","last_fullsized":"2020-11-18T12:01:41.132577+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.132535+0000","last_change":"2020-11-18T11:59:16.174652+0000","last_active":"2020-11-18T12:01:41.132535+0000","last_peered":"2020-11-18T12:01:41.132535+0000","last_clean":"2020-11-18T12:01:41.132535+0000","last_became_active":"2020-11-18T11:59:16.174176+0000","last_became_peered":"2020-11-18T11:59:16.174176+0000","last_unstale":"2020-11-18T12:01:41.132535+0000","last_undegraded":"2020-11-18T12:01:41.132535+0000","last_fullsized":"2020-11-18T12:01:41.132535+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.129724+0000","last_change":"2020-11-18T11:59:16.220833+0000","last_active":"2020-11-18T12:01:41.129724+0000","last_peered":"2020-11-18T12:01:41.129724+0000","last_clean":"2020-11-18T12:01:41.129724+0000","last_became_active":"2020-11-18T11:59:16.220457+0000","last_became_peered":"2020-11-18T11:59:16.220457+0000","last_unstale":"2020-11-18T12:01:41.129724+0000","last_undegraded":"2020-11-18T12:01:41.129724+0000","last_fullsized":"2020-11-18T12:01:41.129724+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":562942753,"num_objects":215,"num_object_clones":0,"num_object_copies":430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":190,"num_whiteouts":0,"num_read":7588,"num_read_kb":3707,"num_write":19465,"num_write_kb":4522598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":25,"num_bytes_hit_set_archive":12609,"num_flush":3426,"num_flush_kb":10013824,"num_evict":3017,"num_evict_kb":8828532,"num_promote":3783,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":509820928,"data_stored":505904942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21454,"ondisk_log_size":21454,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":4,"stat_sum":{"num_bytes":29344083127,"num_objects":9806,"num_object_clones":0,"num_object_copies":19612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9806,"num_whiteouts":0,"num_read":13972,"num_read_kb":107859,"num_write":32777,"num_write_kb":18781443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":37982961664,"data_stored":37838060392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26382,"ondisk_log_size":26382,"up":8,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640295,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9432696,"kb_used_data":9372000,"kb_used_omap":0,"kb_used_meta":51776,"kb_avail":84939144,"statfs":{"total":96636764160,"available":86977683456,"internally_reserved":0,"allocated":9596928000,"data_stored":9559886176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53018624},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.66200000000000003,"15min":0.64900000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60399999999999998,"15min":0.59399999999999997},"min":{"1min":0.41999999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82499999999999996,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.69699999999999995}]},{"osd":1,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67600000000000005,"15min":0.68200000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.77000000000000002,"15min":0.78300000000000003},"min":{"1min":0.51200000000000001,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.202,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.872}]},{"osd":2,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.80600000000000005,"15min":0.79000000000000004},"min":{"1min":0.52000000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.7010000000000001,"5min":1.7010000000000001,"15min":1.7010000000000001},"last":1.1679999999999999},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.86499999999999999,"15min":0.876},"min":{"1min":0.57099999999999995,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":1.026}]},{"osd":3,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.73499999999999999,"15min":0.73299999999999998},"min":{"1min":0.45100000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.80500000000000005,"15min":0.80900000000000005},"min":{"1min":0.56999999999999995,"5min":0.439,"15min":0.439},"max":{"1min":0.98399999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.98399999999999999}]},{"osd":4,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72299999999999998,"15min":0.74299999999999999},"min":{"1min":0.44800000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.875,"15min":0.876},"min":{"1min":0.64400000000000002,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.504,"5min":1.504,"15min":1.504},"last":1.1950000000000001}]}]},{"osd":4,"up_from":12,"seq":51539607592,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155880,"kb_used_data":119456,"kb_used_omap":0,"kb_used_meta":36416,"kb_avail":94215960,"statfs":{"total":96636764160,"available":96477143040,"internally_reserved":0,"allocated":122322944,"data_stored":121186256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37289984},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.68500000000000005,"15min":0.70599999999999996},"min":{"1min":0.40000000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.81499999999999995,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.64100000000000001,"15min":0.65200000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.878,"5min":1.131,"15min":1.131},"last":0.65300000000000002}]},{"osd":1,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.70099999999999996,"15min":0.70999999999999996},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.97499999999999998,"5min":1.145,"15min":1.145},"last":0.746},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.75700000000000001,"15min":0.77300000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.95599999999999996,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.68700000000000006}]},{"osd":2,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.81999999999999995,"15min":0.83799999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.02,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.84499999999999997,"15min":0.86099999999999999},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.054,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.879}]},{"osd":3,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70099999999999996,"15min":0.70999999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.78500000000000003,"15min":0.78900000000000003},"min":{"1min":0.57999999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.93600000000000005,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.94999999999999996}]},{"osd":5,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72799999999999998,"15min":0.72899999999999998},"min":{"1min":0.57599999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.873,"15min":0.88300000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.0349999999999999,"5min":1.264,"15min":1.264},"last":1.0209999999999999}]}]},{"osd":3,"up_from":12,"seq":51539607591,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4595176,"kb_used_data":4564824,"kb_used_omap":0,"kb_used_meta":28864,"kb_avail":89776664,"statfs":{"total":96636764160,"available":91931303936,"internally_reserved":0,"allocated":4674379776,"data_stored":4655592721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29556736},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.59699999999999998,"15min":0.58699999999999997},"min":{"1min":0.38400000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.421,"5min":1.421,"15min":1.421},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63,"15min":0.63500000000000001},"min":{"1min":0.45000000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91700000000000004,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.89200000000000002}]},{"osd":1,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.68700000000000006,"15min":0.68400000000000005},"min":{"1min":0.496,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.51700000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.7030000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":0.85599999999999998}]},{"osd":2,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.503,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.81399999999999995,"15min":0.83499999999999996},"min":{"1min":0.56599999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":1.0209999999999999}]},{"osd":4,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.46100000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.747},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.78000000000000003,"15min":0.79800000000000004},"min":{"1min":0.49099999999999999,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.046,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.93100000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.82899999999999996,"15min":0.80300000000000005},"min":{"1min":0.56399999999999995,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.137},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.83899999999999997,"15min":0.81499999999999995},"min":{"1min":0.65400000000000003,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.3199999999999998,"5min":2.3199999999999998,"15min":2.3199999999999998},"last":0.997}]}]},{"osd":2,"up_from":12,"seq":51539607591,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9531084,"kb_used_data":9470512,"kb_used_omap":0,"kb_used_meta":51328,"kb_avail":84840756,"statfs":{"total":96636764160,"available":86876934144,"internally_reserved":0,"allocated":9697804288,"data_stored":9661420712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52559872},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.502,"15min":0.5},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.438},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.62,"15min":0.622},"min":{"1min":0.40899999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93799999999999994,"5min":1.464,"15min":1.464},"last":0.48799999999999999}]},{"osd":1,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.68999999999999995,"15min":0.68899999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56799999999999995,"15min":0.57399999999999995},"min":{"1min":0.44500000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.67800000000000005,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.53300000000000003}]},{"osd":3,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.69299999999999995,"15min":0.69799999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0069999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.77000000000000002,"15min":0.78000000000000003},"min":{"1min":0.54300000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.026,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.79200000000000004}]},{"osd":4,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.89400000000000002,"15min":0.89800000000000002},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.101,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.876,"15min":0.88600000000000001},"min":{"1min":0.58899999999999997,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.294,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.75700000000000001}]},{"osd":5,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":2.2210000000000001,"15min":2.4700000000000002},"min":{"1min":0.60699999999999998,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.212,"5min":34.686999999999998,"15min":34.686999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.79800000000000004,"5min":2.266,"15min":2.5110000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.0029999999999999,"5min":34.798000000000002,"15min":34.798000000000002},"last":0.71999999999999997}]}]},{"osd":0,"up_from":12,"seq":51539607591,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155880,"kb_used_data":119456,"kb_used_omap":0,"kb_used_meta":36416,"kb_avail":94215960,"statfs":{"total":96636764160,"available":96477143040,"internally_reserved":0,"allocated":122322944,"data_stored":121186256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37289984},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.64000000000000001,"15min":0.65900000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0529999999999999,"5min":1.754,"15min":1.754},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.57999999999999996,"15min":0.56799999999999995},"min":{"1min":0.41999999999999998,"5min":0.37,"15min":0.37},"max":{"1min":1.639,"5min":1.639,"15min":1.639},"last":0.64100000000000001}]},{"osd":2,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.71499999999999997,"15min":0.72199999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.995,"5min":2.4079999999999999,"15min":2.4079999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.65200000000000002,"15min":0.64700000000000002},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.504}]},{"osd":3,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.76600000000000001,"15min":0.77400000000000002},"min":{"1min":0.51800000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1899999999999999,"5min":1.218,"15min":1.218},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.77500000000000002,"15min":0.79200000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.93600000000000005,"5min":1.181,"15min":1.181},"last":0.60099999999999998}]},{"osd":4,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.89400000000000002,"15min":0.91000000000000003},"min":{"1min":0.56799999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.069,"5min":1.9790000000000001,"15min":1.9790000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.91500000000000004,"15min":0.92700000000000005},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.3200000000000001,"5min":2.141,"15min":2.141},"last":0.92700000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.88700000000000001,"15min":0.88800000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.98699999999999999,"15min":1.002},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.276,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":0.79900000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640296,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14022532,"kb_used_data":13946624,"kb_used_omap":0,"kb_used_meta":73344,"kb_avail":80349308,"statfs":{"total":96636764160,"available":82277691392,"internally_reserved":0,"allocated":14281342976,"data_stored":14225990423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":75104256},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.58699999999999997,"15min":0.58099999999999996},"min":{"1min":0.40799999999999997,"5min":0.373,"15min":0.373},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.56899999999999995,"15min":0.55900000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.68400000000000005}]},{"osd":2,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57799999999999996,"15min":0.57699999999999996},"min":{"1min":0.41599999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.746,"5min":0.70199999999999996,"15min":0.69499999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.115,"5min":1.179,"15min":1.179},"last":0.61499999999999999}]},{"osd":3,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.753,"15min":0.76200000000000001},"min":{"1min":0.46300000000000002,"5min":0.441,"15min":0.441},"max":{"1min":1.1040000000000001,"5min":1.139,"15min":1.139},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.80000000000000004,"15min":0.80200000000000005},"min":{"1min":0.48099999999999998,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.96999999999999997}]},{"osd":4,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.90500000000000003,"15min":0.91700000000000004},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.0700000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":1.117},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.88900000000000001,"15min":0.89800000000000002},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.244,"5min":1.536,"15min":1.536},"last":0.93200000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:02:13 2020","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.93700000000000006,"15min":0.93200000000000005},"min":{"1min":0.70199999999999996,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.387,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.0249999999999999,"5min":0.95299999999999996,"15min":0.94099999999999995},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.48,"5min":1.48,"15min":1.48},"last":1.077}]}]}]}} , 2020-11-18T12:02:21.319 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-11-18T11:59:15.163082+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=11, tm_min=59, tm_sec=15, tm_wday=2, tm_yday=323, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=12, tm_min=2, tm_sec=16, tm_wday=2, tm_yday=323, tm_isdst=0) , 2020-11-18T12:02:21.319 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-11-18T11:59:15.163082+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=11, tm_min=59, tm_sec=15, tm_wday=2, tm_yday=323, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=12, tm_min=2, tm_sec=16, tm_wday=2, tm_yday=323, tm_isdst=0) 2020-11-18T18:42:47.402 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-18T12:03:26.330 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi045.front.sepia.ceph.com , 2020-11-18T12:03:26.330 INFO:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 , 2020-11-18T12:03:26.411 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi072.front.sepia.ceph.com 2020-11-18T18:42:47.403 WARNING:scrape:Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' , 2020-11-18T12:05:24.383 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2020-11-18T18:42:47.403 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status 2020-11-18T18:42:47.510 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-7273-gfbf83bb3 (fbf83bb377e36caf429a163721492f1b1c08deb9) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f14e3a605d4] , 2: (DeleteOp::_begin()+0x519) [0x55ce262f4099] 2020-11-18T18:42:47.512 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-18T12:02:21.315 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2020-11-18T12:02:20.101473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29907025880,"num_objects":10021,"num_object_clones":0,"num_object_copies":20042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9996,"num_whiteouts":0,"num_read":21560,"num_read_kb":111566,"num_write":52242,"num_write_kb":23304041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":25,"num_bytes_hit_set_archive":12609,"num_flush":3426,"num_flush_kb":10013824,"num_evict":3017,"num_evict_kb":8828532,"num_promote":3783,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47836,"ondisk_log_size":47836,"up":32,"acting":32,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":32,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":37893248,"kb_used_data":37592872,"kb_used_omap":0,"kb_used_meta":278144,"kb_avail":528337792,"statfs":{"total":579820584960,"available":541017899008,"internally_reserved":0,"allocated":38495100928,"data_stored":38345262544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":284819456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1626810387,"num_objects":537,"num_object_clones":0,"num_object_copies":1074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":537,"num_whiteouts":0,"num_read":1173,"num_read_kb":102087,"num_write":2624,"num_write_kb":930886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":63,"num_flush_kb":174864,"num_evict":65,"num_evict_kb":181271,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":-2,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"24","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130732+0000","last_change":"2020-11-18T11:59:16.220885+0000","last_active":"2020-11-18T12:01:41.130732+0000","last_peered":"2020-11-18T12:01:41.130732+0000","last_clean":"2020-11-18T12:01:41.130732+0000","last_became_active":"2020-11-18T11:59:16.220460+0000","last_became_peered":"2020-11-18T11:59:16.220460+0000","last_unstale":"2020-11-18T12:01:41.130732+0000","last_undegraded":"2020-11-18T12:01:41.130732+0000","last_fullsized":"2020-11-18T12:01:41.130732+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"23","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130021+0000","last_change":"2020-11-18T11:59:16.499247+0000","last_active":"2020-11-18T12:01:41.130021+0000","last_peered":"2020-11-18T12:01:41.130021+0000","last_clean":"2020-11-18T12:01:41.130021+0000","last_became_active":"2020-11-18T11:59:16.498717+0000","last_became_peered":"2020-11-18T11:59:16.498717+0000","last_unstale":"2020-11-18T12:01:41.130021+0000","last_undegraded":"2020-11-18T12:01:41.130021+0000","last_fullsized":"2020-11-18T12:01:41.130021+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"34'5488","reported_seq":"7600","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:05.424086+0000","last_change":"2020-11-18T11:59:27.193656+0000","last_active":"2020-11-18T12:02:05.424086+0000","last_peered":"2020-11-18T12:02:05.424086+0000","last_clean":"2020-11-18T12:02:05.424086+0000","last_became_active":"2020-11-18T11:59:27.193348+0000","last_became_peered":"2020-11-18T11:59:27.193348+0000","last_unstale":"2020-11-18T12:02:05.424086+0000","last_undegraded":"2020-11-18T12:02:05.424086+0000","last_fullsized":"2020-11-18T12:02:05.424086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5488,"ondisk_log_size":5488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":133229742,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":1879,"num_read_kb":944,"num_write":4937,"num_write_kb":1137382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":2934,"num_flush":871,"num_flush_kb":2574747,"num_evict":807,"num_evict_kb":2383573,"num_promote":973,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"34'6592","reported_seq":"8537","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.465679+0000","last_change":"2020-11-18T11:59:25.400522+0000","last_active":"2020-11-18T12:02:15.465679+0000","last_peered":"2020-11-18T12:02:15.465679+0000","last_clean":"2020-11-18T12:02:15.465679+0000","last_became_active":"2020-11-18T11:59:25.399994+0000","last_became_peered":"2020-11-18T11:59:25.399994+0000","last_unstale":"2020-11-18T12:02:15.465679+0000","last_undegraded":"2020-11-18T12:02:15.465679+0000","last_fullsized":"2020-11-18T12:02:15.465679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6592,"ondisk_log_size":6592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7292778019,"num_objects":2447,"num_object_clones":0,"num_object_copies":4894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2447,"num_whiteouts":0,"num_read":3524,"num_read_kb":41917,"num_write":8202,"num_write_kb":4621789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"24","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130666+0000","last_change":"2020-11-18T11:59:16.498778+0000","last_active":"2020-11-18T12:01:41.130666+0000","last_peered":"2020-11-18T12:01:41.130666+0000","last_clean":"2020-11-18T12:01:41.130666+0000","last_became_active":"2020-11-18T11:59:16.498407+0000","last_became_peered":"2020-11-18T11:59:16.498407+0000","last_unstale":"2020-11-18T12:01:41.130666+0000","last_undegraded":"2020-11-18T12:01:41.130666+0000","last_fullsized":"2020-11-18T12:01:41.130666+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"34'5230","reported_seq":"7236","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:06.598034+0000","last_change":"2020-11-18T11:59:27.469664+0000","last_active":"2020-11-18T12:02:06.598034+0000","last_peered":"2020-11-18T12:02:06.598034+0000","last_clean":"2020-11-18T12:02:06.598034+0000","last_became_active":"2020-11-18T11:59:27.469083+0000","last_became_peered":"2020-11-18T11:59:27.469083+0000","last_unstale":"2020-11-18T12:02:06.598034+0000","last_undegraded":"2020-11-18T12:02:06.598034+0000","last_fullsized":"2020-11-18T12:02:06.598034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5230,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":139283403,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":1837,"num_read_kb":905,"num_write":4791,"num_write_kb":1124747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":3402,"num_flush":843,"num_flush_kb":2441950,"num_evict":772,"num_evict_kb":2231779,"num_promote":919,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"34'6685","reported_seq":"8719","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.457184+0000","last_change":"2020-11-18T11:59:25.186658+0000","last_active":"2020-11-18T12:02:15.457184+0000","last_peered":"2020-11-18T12:02:15.457184+0000","last_clean":"2020-11-18T12:02:15.457184+0000","last_became_active":"2020-11-18T11:59:25.186414+0000","last_became_peered":"2020-11-18T11:59:25.186414+0000","last_unstale":"2020-11-18T12:02:15.457184+0000","last_undegraded":"2020-11-18T12:02:15.457184+0000","last_fullsized":"2020-11-18T12:02:15.457184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6685,"ondisk_log_size":6685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7453896300,"num_objects":2479,"num_object_clones":0,"num_object_copies":4958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2479,"num_whiteouts":0,"num_read":3574,"num_read_kb":19412,"num_write":8297,"num_write_kb":4811913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"23","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130006+0000","last_change":"2020-11-18T11:59:16.499200+0000","last_active":"2020-11-18T12:01:41.130006+0000","last_peered":"2020-11-18T12:01:41.130006+0000","last_clean":"2020-11-18T12:01:41.130006+0000","last_became_active":"2020-11-18T11:59:16.498419+0000","last_became_peered":"2020-11-18T11:59:16.498419+0000","last_unstale":"2020-11-18T12:01:41.130006+0000","last_undegraded":"2020-11-18T12:01:41.130006+0000","last_fullsized":"2020-11-18T12:01:41.130006+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"34'5183","reported_seq":"7371","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:06.410297+0000","last_change":"2020-11-18T11:59:27.469738+0000","last_active":"2020-11-18T12:02:06.410297+0000","last_peered":"2020-11-18T12:02:06.410297+0000","last_clean":"2020-11-18T12:02:06.410297+0000","last_became_active":"2020-11-18T11:59:27.469241+0000","last_became_peered":"2020-11-18T11:59:27.469241+0000","last_unstale":"2020-11-18T12:02:06.410297+0000","last_undegraded":"2020-11-18T12:02:06.410297+0000","last_fullsized":"2020-11-18T12:02:06.410297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5183,"ondisk_log_size":5183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":150727528,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":1927,"num_read_kb":886,"num_write":4708,"num_write_kb":1109010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":2455,"num_flush":821,"num_flush_kb":2396213,"num_evict":630,"num_evict_kb":1841924,"num_promote":896,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"34'6480","reported_seq":"8496","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.440474+0000","last_change":"2020-11-18T11:59:25.400594+0000","last_active":"2020-11-18T12:02:15.440474+0000","last_peered":"2020-11-18T12:02:15.440474+0000","last_clean":"2020-11-18T12:02:15.440474+0000","last_became_active":"2020-11-18T11:59:25.400132+0000","last_became_peered":"2020-11-18T11:59:25.400132+0000","last_unstale":"2020-11-18T12:02:15.440474+0000","last_undegraded":"2020-11-18T12:02:15.440474+0000","last_fullsized":"2020-11-18T12:02:15.440474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6480,"ondisk_log_size":6480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7283473264,"num_objects":2438,"num_object_clones":0,"num_object_copies":4876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":3498,"num_read_kb":27406,"num_write":8030,"num_write_kb":4749930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.129702+0000","last_change":"2020-11-18T11:59:16.499029+0000","last_active":"2020-11-18T12:01:41.129702+0000","last_peered":"2020-11-18T12:01:41.129702+0000","last_clean":"2020-11-18T12:01:41.129702+0000","last_became_active":"2020-11-18T11:59:16.498581+0000","last_became_peered":"2020-11-18T11:59:16.498581+0000","last_unstale":"2020-11-18T12:01:41.129702+0000","last_undegraded":"2020-11-18T12:01:41.129702+0000","last_fullsized":"2020-11-18T12:01:41.129702+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"34'5553","reported_seq":"7740","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:07.814557+0000","last_change":"2020-11-18T11:59:27.192743+0000","last_active":"2020-11-18T12:02:07.814557+0000","last_peered":"2020-11-18T12:02:07.814557+0000","last_clean":"2020-11-18T12:02:07.814557+0000","last_became_active":"2020-11-18T11:59:27.192298+0000","last_became_peered":"2020-11-18T11:59:27.192298+0000","last_unstale":"2020-11-18T12:02:07.814557+0000","last_undegraded":"2020-11-18T12:02:07.814557+0000","last_fullsized":"2020-11-18T12:02:07.814557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5553,"ondisk_log_size":5553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":139702080,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":1945,"num_read_kb":972,"num_write":5029,"num_write_kb":1151459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":3818,"num_flush":891,"num_flush_kb":2600914,"num_evict":808,"num_evict_kb":2371256,"num_promote":995,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"34'6625","reported_seq":"8418","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.441145+0000","last_change":"2020-11-18T11:59:25.400403+0000","last_active":"2020-11-18T12:02:15.441145+0000","last_peered":"2020-11-18T12:02:15.441145+0000","last_clean":"2020-11-18T12:02:15.441145+0000","last_became_active":"2020-11-18T11:59:25.400017+0000","last_became_peered":"2020-11-18T11:59:25.400017+0000","last_unstale":"2020-11-18T12:02:15.441145+0000","last_undegraded":"2020-11-18T12:02:15.441145+0000","last_fullsized":"2020-11-18T12:02:15.441145+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6625,"ondisk_log_size":6625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7313935544,"num_objects":2442,"num_object_clones":0,"num_object_copies":4884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2442,"num_whiteouts":0,"num_read":3376,"num_read_kb":19124,"num_write":8248,"num_write_kb":4597811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.132577+0000","last_change":"2020-11-18T11:59:16.174660+0000","last_active":"2020-11-18T12:01:41.132577+0000","last_peered":"2020-11-18T12:01:41.132577+0000","last_clean":"2020-11-18T12:01:41.132577+0000","last_became_active":"2020-11-18T11:59:16.174201+0000","last_became_peered":"2020-11-18T11:59:16.174201+0000","last_unstale":"2020-11-18T12:01:41.132577+0000","last_undegraded":"2020-11-18T12:01:41.132577+0000","last_fullsized":"2020-11-18T12:01:41.132577+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.132535+0000","last_change":"2020-11-18T11:59:16.174652+0000","last_active":"2020-11-18T12:01:41.132535+0000","last_peered":"2020-11-18T12:01:41.132535+0000","last_clean":"2020-11-18T12:01:41.132535+0000","last_became_active":"2020-11-18T11:59:16.174176+0000","last_became_peered":"2020-11-18T11:59:16.174176+0000","last_unstale":"2020-11-18T12:01:41.132535+0000","last_undegraded":"2020-11-18T12:01:41.132535+0000","last_fullsized":"2020-11-18T12:01:41.132535+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.129724+0000","last_change":"2020-11-18T11:59:16.220833+0000","last_active":"2020-11-18T12:01:41.129724+0000","last_peered":"2020-11-18T12:01:41.129724+0000","last_clean":"2020-11-18T12:01:41.129724+0000","last_became_active":"2020-11-18T11:59:16.220457+0000","last_became_peered":"2020-11-18T11:59:16.220457+0000","last_unstale":"2020-11-18T12:01:41.129724+0000","last_undegraded":"2020-11-18T12:01:41.129724+0000","last_fullsized":"2020-11-18T12:01:41.129724+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":562942753,"num_objects":215,"num_object_clones":0,"num_object_copies":430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":190,"num_whiteouts":0,"num_read":7588,"num_read_kb":3707,"num_write":19465,"num_write_kb":4522598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":25,"num_bytes_hit_set_archive":12609,"num_flush":3426,"num_flush_kb":10013824,"num_evict":3017,"num_evict_kb":8828532,"num_promote":3783,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":509820928,"data_stored":505904942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21454,"ondisk_log_size":21454,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":4,"stat_sum":{"num_bytes":29344083127,"num_objects":9806,"num_object_clones":0,"num_object_copies":19612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9806,"num_whiteouts":0,"num_read":13972,"num_read_kb":107859,"num_write":32777,"num_write_kb":18781443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":37982961664,"data_stored":37838060392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26382,"ondisk_log_size":26382,"up":8,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640295,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9432696,"kb_used_data":9372000,"kb_used_omap":0,"kb_used_meta":51776,"kb_avail":84939144,"statfs":{"total":96636764160,"available":86977683456,"internally_reserved":0,"allocated":9596928000,"data_stored":9559886176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53018624},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.66200000000000003,"15min":0.64900000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60399999999999998,"15min":0.59399999999999997},"min":{"1min":0.41999999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82499999999999996,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.69699999999999995}]},{"osd":1,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67600000000000005,"15min":0.68200000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.77000000000000002,"15min":0.78300000000000003},"min":{"1min":0.51200000000000001,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.202,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.872}]},{"osd":2,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.80600000000000005,"15min":0.79000000000000004},"min":{"1min":0.52000000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.7010000000000001,"5min":1.7010000000000001,"15min":1.7010000000000001},"last":1.1679999999999999},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.86499999999999999,"15min":0.876},"min":{"1min":0.57099999999999995,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":1.026}]},{"osd":3,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.73499999999999999,"15min":0.73299999999999998},"min":{"1min":0.45100000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.80500000000000005,"15min":0.80900000000000005},"min":{"1min":0.56999999999999995,"5min":0.439,"15min":0.439},"max":{"1min":0.98399999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.98399999999999999}]},{"osd":4,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72299999999999998,"15min":0.74299999999999999},"min":{"1min":0.44800000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.875,"15min":0.876},"min":{"1min":0.64400000000000002,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.504,"5min":1.504,"15min":1.504},"last":1.1950000000000001}]}]},{"osd":4,"up_from":12,"seq":51539607592,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155880,"kb_used_data":119456,"kb_used_omap":0,"kb_used_meta":36416,"kb_avail":94215960,"statfs":{"total":96636764160,"available":96477143040,"internally_reserved":0,"allocated":122322944,"data_stored":121186256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37289984},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.68500000000000005,"15min":0.70599999999999996},"min":{"1min":0.40000000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.81499999999999995,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.64100000000000001,"15min":0.65200000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.878,"5min":1.131,"15min":1.131},"last":0.65300000000000002}]},{"osd":1,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.70099999999999996,"15min":0.70999999999999996},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.97499999999999998,"5min":1.145,"15min":1.145},"last":0.746},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.75700000000000001,"15min":0.77300000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.95599999999999996,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.68700000000000006}]},{"osd":2,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.81999999999999995,"15min":0.83799999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.02,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.84499999999999997,"15min":0.86099999999999999},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.054,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.879}]},{"osd":3,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70099999999999996,"15min":0.70999999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.78500000000000003,"15min":0.78900000000000003},"min":{"1min":0.57999999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.93600000000000005,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.94999999999999996}]},{"osd":5,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72799999999999998,"15min":0.72899999999999998},"min":{"1min":0.57599999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.873,"15min":0.88300000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.0349999999999999,"5min":1.264,"15min":1.264},"last":1.0209999999999999}]}]},{"osd":3,"up_from":12,"seq":51539607591,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4595176,"kb_used_data":4564824,"kb_used_omap":0,"kb_used_meta":28864,"kb_avail":89776664,"statfs":{"total":96636764160,"available":91931303936,"internally_reserved":0,"allocated":4674379776,"data_stored":4655592721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29556736},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.59699999999999998,"15min":0.58699999999999997},"min":{"1min":0.38400000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.421,"5min":1.421,"15min":1.421},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63,"15min":0.63500000000000001},"min":{"1min":0.45000000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91700000000000004,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.89200000000000002}]},{"osd":1,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.68700000000000006,"15min":0.68400000000000005},"min":{"1min":0.496,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.51700000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.7030000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":0.85599999999999998}]},{"osd":2,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.503,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.81399999999999995,"15min":0.83499999999999996},"min":{"1min":0.56599999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":1.0209999999999999}]},{"osd":4,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.46100000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.747},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.78000000000000003,"15min":0.79800000000000004},"min":{"1min":0.49099999999999999,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.046,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.93100000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.82899999999999996,"15min":0.80300000000000005},"min":{"1min":0.56399999999999995,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.137},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.83899999999999997,"15min":0.81499999999999995},"min":{"1min":0.65400000000000003,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.3199999999999998,"5min":2.3199999999999998,"15min":2.3199999999999998},"last":0.997}]}]},{"osd":2,"up_from":12,"seq":51539607591,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9531084,"kb_used_data":9470512,"kb_used_omap":0,"kb_used_meta":51328,"kb_avail":84840756,"statfs":{"total":96636764160,"available":86876934144,"internally_reserved":0,"allocated":9697804288,"data_stored":9661420712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52559872},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.502,"15min":0.5},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.438},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.62,"15min":0.622},"min":{"1min":0.40899999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93799999999999994,"5min":1.464,"15min":1.464},"last":0.48799999999999999}]},{"osd":1,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.68999999999999995,"15min":0.68899999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56799999999999995,"15min":0.57399999999999995},"min":{"1min":0.44500000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.67800000000000005,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.53300000000000003}]},{"osd":3,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.69299999999999995,"15min":0.69799999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0069999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.77000000000000002,"15min":0.78000000000000003},"min":{"1min":0.54300000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.026,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.79200000000000004}]},{"osd":4,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.89400000000000002,"15min":0.89800000000000002},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.101,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.876,"15min":0.88600000000000001},"min":{"1min":0.58899999999999997,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.294,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.75700000000000001}]},{"osd":5,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":2.2210000000000001,"15min":2.4700000000000002},"min":{"1min":0.60699999999999998,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.212,"5min":34.686999999999998,"15min":34.686999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.79800000000000004,"5min":2.266,"15min":2.5110000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.0029999999999999,"5min":34.798000000000002,"15min":34.798000000000002},"last":0.71999999999999997}]}]},{"osd":0,"up_from":12,"seq":51539607591,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155880,"kb_used_data":119456,"kb_used_omap":0,"kb_used_meta":36416,"kb_avail":94215960,"statfs":{"total":96636764160,"available":96477143040,"internally_reserved":0,"allocated":122322944,"data_stored":121186256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37289984},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.64000000000000001,"15min":0.65900000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0529999999999999,"5min":1.754,"15min":1.754},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.57999999999999996,"15min":0.56799999999999995},"min":{"1min":0.41999999999999998,"5min":0.37,"15min":0.37},"max":{"1min":1.639,"5min":1.639,"15min":1.639},"last":0.64100000000000001}]},{"osd":2,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.71499999999999997,"15min":0.72199999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.995,"5min":2.4079999999999999,"15min":2.4079999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.65200000000000002,"15min":0.64700000000000002},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.504}]},{"osd":3,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.76600000000000001,"15min":0.77400000000000002},"min":{"1min":0.51800000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1899999999999999,"5min":1.218,"15min":1.218},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.77500000000000002,"15min":0.79200000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.93600000000000005,"5min":1.181,"15min":1.181},"last":0.60099999999999998}]},{"osd":4,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.89400000000000002,"15min":0.91000000000000003},"min":{"1min":0.56799999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.069,"5min":1.9790000000000001,"15min":1.9790000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.91500000000000004,"15min":0.92700000000000005},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.3200000000000001,"5min":2.141,"15min":2.141},"last":0.92700000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.88700000000000001,"15min":0.88800000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.98699999999999999,"15min":1.002},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.276,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":0.79900000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640296,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14022532,"kb_used_data":13946624,"kb_used_omap":0,"kb_used_meta":73344,"kb_avail":80349308,"statfs":{"total":96636764160,"available":82277691392,"internally_reserved":0,"allocated":14281342976,"data_stored":14225990423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":75104256},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.58699999999999997,"15min":0.58099999999999996},"min":{"1min":0.40799999999999997,"5min":0.373,"15min":0.373},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.56899999999999995,"15min":0.55900000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.68400000000000005}]},{"osd":2,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57799999999999996,"15min":0.57699999999999996},"min":{"1min":0.41599999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.746,"5min":0.70199999999999996,"15min":0.69499999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.115,"5min":1.179,"15min":1.179},"last":0.61499999999999999}]},{"osd":3,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.753,"15min":0.76200000000000001},"min":{"1min":0.46300000000000002,"5min":0.441,"15min":0.441},"max":{"1min":1.1040000000000001,"5min":1.139,"15min":1.139},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.80000000000000004,"15min":0.80200000000000005},"min":{"1min":0.48099999999999998,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.96999999999999997}]},{"osd":4,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.90500000000000003,"15min":0.91700000000000004},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.0700000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":1.117},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.88900000000000001,"15min":0.89800000000000002},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.244,"5min":1.536,"15min":1.536},"last":0.93200000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:02:13 2020","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.93700000000000006,"15min":0.93200000000000005},"min":{"1min":0.70199999999999996,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.387,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.0249999999999999,"5min":0.95299999999999996,"15min":0.94099999999999995},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.48,"5min":1.48,"15min":1.48},"last":1.077}]}]}]}} , 2020-11-18T12:02:21.319 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-11-18T11:59:15.163082+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=11, tm_min=59, tm_sec=15, tm_wday=2, tm_yday=323, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=12, tm_min=2, tm_sec=16, tm_wday=2, tm_yday=323, tm_isdst=0) , 2020-11-18T12:02:21.319 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-11-18T11:59:15.163082+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=11, tm_min=59, tm_sec=15, tm_wday=2, tm_yday=323, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=12, tm_min=2, tm_sec=16, tm_wday=2, tm_yday=323, tm_isdst=0) 2020-11-18T18:42:47.514 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-18T12:03:26.330 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi045.front.sepia.ceph.com , 2020-11-18T12:03:26.330 INFO:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 , 2020-11-18T12:03:26.411 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi072.front.sepia.ceph.com 2020-11-18T18:42:47.515 WARNING:scrape:Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' , 2020-11-18T12:05:24.383 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2020-11-18T18:42:47.515 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status 2020-11-18T18:42:47.618 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-7273-gfbf83bb3 (fbf83bb377e36caf429a163721492f1b1c08deb9) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f14e3a605d4] , 2: (DeleteOp::_begin()+0x519) [0x55ce262f4099] 2020-11-18T18:42:47.620 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-18T12:02:21.315 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2020-11-18T12:02:20.101473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29907025880,"num_objects":10021,"num_object_clones":0,"num_object_copies":20042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9996,"num_whiteouts":0,"num_read":21560,"num_read_kb":111566,"num_write":52242,"num_write_kb":23304041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":25,"num_bytes_hit_set_archive":12609,"num_flush":3426,"num_flush_kb":10013824,"num_evict":3017,"num_evict_kb":8828532,"num_promote":3783,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47836,"ondisk_log_size":47836,"up":32,"acting":32,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":32,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":37893248,"kb_used_data":37592872,"kb_used_omap":0,"kb_used_meta":278144,"kb_avail":528337792,"statfs":{"total":579820584960,"available":541017899008,"internally_reserved":0,"allocated":38495100928,"data_stored":38345262544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":284819456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1626810387,"num_objects":537,"num_object_clones":0,"num_object_copies":1074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":537,"num_whiteouts":0,"num_read":1173,"num_read_kb":102087,"num_write":2624,"num_write_kb":930886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":63,"num_flush_kb":174864,"num_evict":65,"num_evict_kb":181271,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":-2,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"24","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130732+0000","last_change":"2020-11-18T11:59:16.220885+0000","last_active":"2020-11-18T12:01:41.130732+0000","last_peered":"2020-11-18T12:01:41.130732+0000","last_clean":"2020-11-18T12:01:41.130732+0000","last_became_active":"2020-11-18T11:59:16.220460+0000","last_became_peered":"2020-11-18T11:59:16.220460+0000","last_unstale":"2020-11-18T12:01:41.130732+0000","last_undegraded":"2020-11-18T12:01:41.130732+0000","last_fullsized":"2020-11-18T12:01:41.130732+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"23","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130021+0000","last_change":"2020-11-18T11:59:16.499247+0000","last_active":"2020-11-18T12:01:41.130021+0000","last_peered":"2020-11-18T12:01:41.130021+0000","last_clean":"2020-11-18T12:01:41.130021+0000","last_became_active":"2020-11-18T11:59:16.498717+0000","last_became_peered":"2020-11-18T11:59:16.498717+0000","last_unstale":"2020-11-18T12:01:41.130021+0000","last_undegraded":"2020-11-18T12:01:41.130021+0000","last_fullsized":"2020-11-18T12:01:41.130021+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"34'5488","reported_seq":"7600","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:05.424086+0000","last_change":"2020-11-18T11:59:27.193656+0000","last_active":"2020-11-18T12:02:05.424086+0000","last_peered":"2020-11-18T12:02:05.424086+0000","last_clean":"2020-11-18T12:02:05.424086+0000","last_became_active":"2020-11-18T11:59:27.193348+0000","last_became_peered":"2020-11-18T11:59:27.193348+0000","last_unstale":"2020-11-18T12:02:05.424086+0000","last_undegraded":"2020-11-18T12:02:05.424086+0000","last_fullsized":"2020-11-18T12:02:05.424086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5488,"ondisk_log_size":5488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":133229742,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":1879,"num_read_kb":944,"num_write":4937,"num_write_kb":1137382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":2934,"num_flush":871,"num_flush_kb":2574747,"num_evict":807,"num_evict_kb":2383573,"num_promote":973,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"34'6592","reported_seq":"8537","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.465679+0000","last_change":"2020-11-18T11:59:25.400522+0000","last_active":"2020-11-18T12:02:15.465679+0000","last_peered":"2020-11-18T12:02:15.465679+0000","last_clean":"2020-11-18T12:02:15.465679+0000","last_became_active":"2020-11-18T11:59:25.399994+0000","last_became_peered":"2020-11-18T11:59:25.399994+0000","last_unstale":"2020-11-18T12:02:15.465679+0000","last_undegraded":"2020-11-18T12:02:15.465679+0000","last_fullsized":"2020-11-18T12:02:15.465679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6592,"ondisk_log_size":6592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7292778019,"num_objects":2447,"num_object_clones":0,"num_object_copies":4894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2447,"num_whiteouts":0,"num_read":3524,"num_read_kb":41917,"num_write":8202,"num_write_kb":4621789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"24","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130666+0000","last_change":"2020-11-18T11:59:16.498778+0000","last_active":"2020-11-18T12:01:41.130666+0000","last_peered":"2020-11-18T12:01:41.130666+0000","last_clean":"2020-11-18T12:01:41.130666+0000","last_became_active":"2020-11-18T11:59:16.498407+0000","last_became_peered":"2020-11-18T11:59:16.498407+0000","last_unstale":"2020-11-18T12:01:41.130666+0000","last_undegraded":"2020-11-18T12:01:41.130666+0000","last_fullsized":"2020-11-18T12:01:41.130666+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"34'5230","reported_seq":"7236","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:06.598034+0000","last_change":"2020-11-18T11:59:27.469664+0000","last_active":"2020-11-18T12:02:06.598034+0000","last_peered":"2020-11-18T12:02:06.598034+0000","last_clean":"2020-11-18T12:02:06.598034+0000","last_became_active":"2020-11-18T11:59:27.469083+0000","last_became_peered":"2020-11-18T11:59:27.469083+0000","last_unstale":"2020-11-18T12:02:06.598034+0000","last_undegraded":"2020-11-18T12:02:06.598034+0000","last_fullsized":"2020-11-18T12:02:06.598034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5230,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":139283403,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":1837,"num_read_kb":905,"num_write":4791,"num_write_kb":1124747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":3402,"num_flush":843,"num_flush_kb":2441950,"num_evict":772,"num_evict_kb":2231779,"num_promote":919,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"34'6685","reported_seq":"8719","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.457184+0000","last_change":"2020-11-18T11:59:25.186658+0000","last_active":"2020-11-18T12:02:15.457184+0000","last_peered":"2020-11-18T12:02:15.457184+0000","last_clean":"2020-11-18T12:02:15.457184+0000","last_became_active":"2020-11-18T11:59:25.186414+0000","last_became_peered":"2020-11-18T11:59:25.186414+0000","last_unstale":"2020-11-18T12:02:15.457184+0000","last_undegraded":"2020-11-18T12:02:15.457184+0000","last_fullsized":"2020-11-18T12:02:15.457184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6685,"ondisk_log_size":6685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7453896300,"num_objects":2479,"num_object_clones":0,"num_object_copies":4958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2479,"num_whiteouts":0,"num_read":3574,"num_read_kb":19412,"num_write":8297,"num_write_kb":4811913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"23","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130006+0000","last_change":"2020-11-18T11:59:16.499200+0000","last_active":"2020-11-18T12:01:41.130006+0000","last_peered":"2020-11-18T12:01:41.130006+0000","last_clean":"2020-11-18T12:01:41.130006+0000","last_became_active":"2020-11-18T11:59:16.498419+0000","last_became_peered":"2020-11-18T11:59:16.498419+0000","last_unstale":"2020-11-18T12:01:41.130006+0000","last_undegraded":"2020-11-18T12:01:41.130006+0000","last_fullsized":"2020-11-18T12:01:41.130006+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"34'5183","reported_seq":"7371","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:06.410297+0000","last_change":"2020-11-18T11:59:27.469738+0000","last_active":"2020-11-18T12:02:06.410297+0000","last_peered":"2020-11-18T12:02:06.410297+0000","last_clean":"2020-11-18T12:02:06.410297+0000","last_became_active":"2020-11-18T11:59:27.469241+0000","last_became_peered":"2020-11-18T11:59:27.469241+0000","last_unstale":"2020-11-18T12:02:06.410297+0000","last_undegraded":"2020-11-18T12:02:06.410297+0000","last_fullsized":"2020-11-18T12:02:06.410297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5183,"ondisk_log_size":5183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":150727528,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":1927,"num_read_kb":886,"num_write":4708,"num_write_kb":1109010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":2455,"num_flush":821,"num_flush_kb":2396213,"num_evict":630,"num_evict_kb":1841924,"num_promote":896,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"34'6480","reported_seq":"8496","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.440474+0000","last_change":"2020-11-18T11:59:25.400594+0000","last_active":"2020-11-18T12:02:15.440474+0000","last_peered":"2020-11-18T12:02:15.440474+0000","last_clean":"2020-11-18T12:02:15.440474+0000","last_became_active":"2020-11-18T11:59:25.400132+0000","last_became_peered":"2020-11-18T11:59:25.400132+0000","last_unstale":"2020-11-18T12:02:15.440474+0000","last_undegraded":"2020-11-18T12:02:15.440474+0000","last_fullsized":"2020-11-18T12:02:15.440474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6480,"ondisk_log_size":6480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7283473264,"num_objects":2438,"num_object_clones":0,"num_object_copies":4876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":3498,"num_read_kb":27406,"num_write":8030,"num_write_kb":4749930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.129702+0000","last_change":"2020-11-18T11:59:16.499029+0000","last_active":"2020-11-18T12:01:41.129702+0000","last_peered":"2020-11-18T12:01:41.129702+0000","last_clean":"2020-11-18T12:01:41.129702+0000","last_became_active":"2020-11-18T11:59:16.498581+0000","last_became_peered":"2020-11-18T11:59:16.498581+0000","last_unstale":"2020-11-18T12:01:41.129702+0000","last_undegraded":"2020-11-18T12:01:41.129702+0000","last_fullsized":"2020-11-18T12:01:41.129702+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"34'5553","reported_seq":"7740","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:07.814557+0000","last_change":"2020-11-18T11:59:27.192743+0000","last_active":"2020-11-18T12:02:07.814557+0000","last_peered":"2020-11-18T12:02:07.814557+0000","last_clean":"2020-11-18T12:02:07.814557+0000","last_became_active":"2020-11-18T11:59:27.192298+0000","last_became_peered":"2020-11-18T11:59:27.192298+0000","last_unstale":"2020-11-18T12:02:07.814557+0000","last_undegraded":"2020-11-18T12:02:07.814557+0000","last_fullsized":"2020-11-18T12:02:07.814557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5553,"ondisk_log_size":5553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":139702080,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":1945,"num_read_kb":972,"num_write":5029,"num_write_kb":1151459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":3818,"num_flush":891,"num_flush_kb":2600914,"num_evict":808,"num_evict_kb":2371256,"num_promote":995,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"34'6625","reported_seq":"8418","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.441145+0000","last_change":"2020-11-18T11:59:25.400403+0000","last_active":"2020-11-18T12:02:15.441145+0000","last_peered":"2020-11-18T12:02:15.441145+0000","last_clean":"2020-11-18T12:02:15.441145+0000","last_became_active":"2020-11-18T11:59:25.400017+0000","last_became_peered":"2020-11-18T11:59:25.400017+0000","last_unstale":"2020-11-18T12:02:15.441145+0000","last_undegraded":"2020-11-18T12:02:15.441145+0000","last_fullsized":"2020-11-18T12:02:15.441145+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6625,"ondisk_log_size":6625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7313935544,"num_objects":2442,"num_object_clones":0,"num_object_copies":4884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2442,"num_whiteouts":0,"num_read":3376,"num_read_kb":19124,"num_write":8248,"num_write_kb":4597811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.132577+0000","last_change":"2020-11-18T11:59:16.174660+0000","last_active":"2020-11-18T12:01:41.132577+0000","last_peered":"2020-11-18T12:01:41.132577+0000","last_clean":"2020-11-18T12:01:41.132577+0000","last_became_active":"2020-11-18T11:59:16.174201+0000","last_became_peered":"2020-11-18T11:59:16.174201+0000","last_unstale":"2020-11-18T12:01:41.132577+0000","last_undegraded":"2020-11-18T12:01:41.132577+0000","last_fullsized":"2020-11-18T12:01:41.132577+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.132535+0000","last_change":"2020-11-18T11:59:16.174652+0000","last_active":"2020-11-18T12:01:41.132535+0000","last_peered":"2020-11-18T12:01:41.132535+0000","last_clean":"2020-11-18T12:01:41.132535+0000","last_became_active":"2020-11-18T11:59:16.174176+0000","last_became_peered":"2020-11-18T11:59:16.174176+0000","last_unstale":"2020-11-18T12:01:41.132535+0000","last_undegraded":"2020-11-18T12:01:41.132535+0000","last_fullsized":"2020-11-18T12:01:41.132535+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.129724+0000","last_change":"2020-11-18T11:59:16.220833+0000","last_active":"2020-11-18T12:01:41.129724+0000","last_peered":"2020-11-18T12:01:41.129724+0000","last_clean":"2020-11-18T12:01:41.129724+0000","last_became_active":"2020-11-18T11:59:16.220457+0000","last_became_peered":"2020-11-18T11:59:16.220457+0000","last_unstale":"2020-11-18T12:01:41.129724+0000","last_undegraded":"2020-11-18T12:01:41.129724+0000","last_fullsized":"2020-11-18T12:01:41.129724+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":562942753,"num_objects":215,"num_object_clones":0,"num_object_copies":430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":190,"num_whiteouts":0,"num_read":7588,"num_read_kb":3707,"num_write":19465,"num_write_kb":4522598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":25,"num_bytes_hit_set_archive":12609,"num_flush":3426,"num_flush_kb":10013824,"num_evict":3017,"num_evict_kb":8828532,"num_promote":3783,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":509820928,"data_stored":505904942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21454,"ondisk_log_size":21454,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":4,"stat_sum":{"num_bytes":29344083127,"num_objects":9806,"num_object_clones":0,"num_object_copies":19612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9806,"num_whiteouts":0,"num_read":13972,"num_read_kb":107859,"num_write":32777,"num_write_kb":18781443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":37982961664,"data_stored":37838060392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26382,"ondisk_log_size":26382,"up":8,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640295,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9432696,"kb_used_data":9372000,"kb_used_omap":0,"kb_used_meta":51776,"kb_avail":84939144,"statfs":{"total":96636764160,"available":86977683456,"internally_reserved":0,"allocated":9596928000,"data_stored":9559886176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53018624},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.66200000000000003,"15min":0.64900000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60399999999999998,"15min":0.59399999999999997},"min":{"1min":0.41999999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82499999999999996,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.69699999999999995}]},{"osd":1,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67600000000000005,"15min":0.68200000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.77000000000000002,"15min":0.78300000000000003},"min":{"1min":0.51200000000000001,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.202,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.872}]},{"osd":2,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.80600000000000005,"15min":0.79000000000000004},"min":{"1min":0.52000000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.7010000000000001,"5min":1.7010000000000001,"15min":1.7010000000000001},"last":1.1679999999999999},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.86499999999999999,"15min":0.876},"min":{"1min":0.57099999999999995,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":1.026}]},{"osd":3,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.73499999999999999,"15min":0.73299999999999998},"min":{"1min":0.45100000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.80500000000000005,"15min":0.80900000000000005},"min":{"1min":0.56999999999999995,"5min":0.439,"15min":0.439},"max":{"1min":0.98399999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.98399999999999999}]},{"osd":4,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72299999999999998,"15min":0.74299999999999999},"min":{"1min":0.44800000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.875,"15min":0.876},"min":{"1min":0.64400000000000002,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.504,"5min":1.504,"15min":1.504},"last":1.1950000000000001}]}]},{"osd":4,"up_from":12,"seq":51539607592,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155880,"kb_used_data":119456,"kb_used_omap":0,"kb_used_meta":36416,"kb_avail":94215960,"statfs":{"total":96636764160,"available":96477143040,"internally_reserved":0,"allocated":122322944,"data_stored":121186256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37289984},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.68500000000000005,"15min":0.70599999999999996},"min":{"1min":0.40000000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.81499999999999995,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.64100000000000001,"15min":0.65200000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.878,"5min":1.131,"15min":1.131},"last":0.65300000000000002}]},{"osd":1,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.70099999999999996,"15min":0.70999999999999996},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.97499999999999998,"5min":1.145,"15min":1.145},"last":0.746},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.75700000000000001,"15min":0.77300000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.95599999999999996,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.68700000000000006}]},{"osd":2,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.81999999999999995,"15min":0.83799999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.02,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.84499999999999997,"15min":0.86099999999999999},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.054,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.879}]},{"osd":3,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70099999999999996,"15min":0.70999999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.78500000000000003,"15min":0.78900000000000003},"min":{"1min":0.57999999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.93600000000000005,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.94999999999999996}]},{"osd":5,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72799999999999998,"15min":0.72899999999999998},"min":{"1min":0.57599999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.873,"15min":0.88300000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.0349999999999999,"5min":1.264,"15min":1.264},"last":1.0209999999999999}]}]},{"osd":3,"up_from":12,"seq":51539607591,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4595176,"kb_used_data":4564824,"kb_used_omap":0,"kb_used_meta":28864,"kb_avail":89776664,"statfs":{"total":96636764160,"available":91931303936,"internally_reserved":0,"allocated":4674379776,"data_stored":4655592721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29556736},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.59699999999999998,"15min":0.58699999999999997},"min":{"1min":0.38400000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.421,"5min":1.421,"15min":1.421},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63,"15min":0.63500000000000001},"min":{"1min":0.45000000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91700000000000004,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.89200000000000002}]},{"osd":1,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.68700000000000006,"15min":0.68400000000000005},"min":{"1min":0.496,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.51700000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.7030000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":0.85599999999999998}]},{"osd":2,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.503,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.81399999999999995,"15min":0.83499999999999996},"min":{"1min":0.56599999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":1.0209999999999999}]},{"osd":4,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.46100000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.747},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.78000000000000003,"15min":0.79800000000000004},"min":{"1min":0.49099999999999999,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.046,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.93100000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.82899999999999996,"15min":0.80300000000000005},"min":{"1min":0.56399999999999995,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.137},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.83899999999999997,"15min":0.81499999999999995},"min":{"1min":0.65400000000000003,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.3199999999999998,"5min":2.3199999999999998,"15min":2.3199999999999998},"last":0.997}]}]},{"osd":2,"up_from":12,"seq":51539607591,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9531084,"kb_used_data":9470512,"kb_used_omap":0,"kb_used_meta":51328,"kb_avail":84840756,"statfs":{"total":96636764160,"available":86876934144,"internally_reserved":0,"allocated":9697804288,"data_stored":9661420712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52559872},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.502,"15min":0.5},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.438},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.62,"15min":0.622},"min":{"1min":0.40899999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93799999999999994,"5min":1.464,"15min":1.464},"last":0.48799999999999999}]},{"osd":1,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.68999999999999995,"15min":0.68899999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56799999999999995,"15min":0.57399999999999995},"min":{"1min":0.44500000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.67800000000000005,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.53300000000000003}]},{"osd":3,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.69299999999999995,"15min":0.69799999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0069999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.77000000000000002,"15min":0.78000000000000003},"min":{"1min":0.54300000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.026,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.79200000000000004}]},{"osd":4,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.89400000000000002,"15min":0.89800000000000002},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.101,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.876,"15min":0.88600000000000001},"min":{"1min":0.58899999999999997,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.294,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.75700000000000001}]},{"osd":5,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":2.2210000000000001,"15min":2.4700000000000002},"min":{"1min":0.60699999999999998,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.212,"5min":34.686999999999998,"15min":34.686999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.79800000000000004,"5min":2.266,"15min":2.5110000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.0029999999999999,"5min":34.798000000000002,"15min":34.798000000000002},"last":0.71999999999999997}]}]},{"osd":0,"up_from":12,"seq":51539607591,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155880,"kb_used_data":119456,"kb_used_omap":0,"kb_used_meta":36416,"kb_avail":94215960,"statfs":{"total":96636764160,"available":96477143040,"internally_reserved":0,"allocated":122322944,"data_stored":121186256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37289984},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.64000000000000001,"15min":0.65900000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0529999999999999,"5min":1.754,"15min":1.754},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.57999999999999996,"15min":0.56799999999999995},"min":{"1min":0.41999999999999998,"5min":0.37,"15min":0.37},"max":{"1min":1.639,"5min":1.639,"15min":1.639},"last":0.64100000000000001}]},{"osd":2,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.71499999999999997,"15min":0.72199999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.995,"5min":2.4079999999999999,"15min":2.4079999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.65200000000000002,"15min":0.64700000000000002},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.504}]},{"osd":3,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.76600000000000001,"15min":0.77400000000000002},"min":{"1min":0.51800000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1899999999999999,"5min":1.218,"15min":1.218},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.77500000000000002,"15min":0.79200000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.93600000000000005,"5min":1.181,"15min":1.181},"last":0.60099999999999998}]},{"osd":4,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.89400000000000002,"15min":0.91000000000000003},"min":{"1min":0.56799999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.069,"5min":1.9790000000000001,"15min":1.9790000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.91500000000000004,"15min":0.92700000000000005},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.3200000000000001,"5min":2.141,"15min":2.141},"last":0.92700000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.88700000000000001,"15min":0.88800000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.98699999999999999,"15min":1.002},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.276,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":0.79900000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640296,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14022532,"kb_used_data":13946624,"kb_used_omap":0,"kb_used_meta":73344,"kb_avail":80349308,"statfs":{"total":96636764160,"available":82277691392,"internally_reserved":0,"allocated":14281342976,"data_stored":14225990423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":75104256},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.58699999999999997,"15min":0.58099999999999996},"min":{"1min":0.40799999999999997,"5min":0.373,"15min":0.373},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.56899999999999995,"15min":0.55900000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.68400000000000005}]},{"osd":2,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57799999999999996,"15min":0.57699999999999996},"min":{"1min":0.41599999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.746,"5min":0.70199999999999996,"15min":0.69499999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.115,"5min":1.179,"15min":1.179},"last":0.61499999999999999}]},{"osd":3,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.753,"15min":0.76200000000000001},"min":{"1min":0.46300000000000002,"5min":0.441,"15min":0.441},"max":{"1min":1.1040000000000001,"5min":1.139,"15min":1.139},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.80000000000000004,"15min":0.80200000000000005},"min":{"1min":0.48099999999999998,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.96999999999999997}]},{"osd":4,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.90500000000000003,"15min":0.91700000000000004},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.0700000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":1.117},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.88900000000000001,"15min":0.89800000000000002},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.244,"5min":1.536,"15min":1.536},"last":0.93200000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:02:13 2020","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.93700000000000006,"15min":0.93200000000000005},"min":{"1min":0.70199999999999996,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.387,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.0249999999999999,"5min":0.95299999999999996,"15min":0.94099999999999995},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.48,"5min":1.48,"15min":1.48},"last":1.077}]}]}]}} , 2020-11-18T12:02:21.319 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-11-18T11:59:15.163082+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=11, tm_min=59, tm_sec=15, tm_wday=2, tm_yday=323, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=12, tm_min=2, tm_sec=16, tm_wday=2, tm_yday=323, tm_isdst=0) , 2020-11-18T12:02:21.319 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-11-18T11:59:15.163082+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=11, tm_min=59, tm_sec=15, tm_wday=2, tm_yday=323, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=12, tm_min=2, tm_sec=16, tm_wday=2, tm_yday=323, tm_isdst=0) 2020-11-18T18:42:47.622 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-18T12:03:26.330 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi045.front.sepia.ceph.com , 2020-11-18T12:03:26.330 INFO:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 , 2020-11-18T12:03:26.411 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi072.front.sepia.ceph.com 2020-11-18T18:42:47.622 WARNING:scrape:Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' , 2020-11-18T12:05:24.383 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2020-11-18T18:42:47.623 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status 2020-11-18T18:42:47.728 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-7273-gfbf83bb3 (fbf83bb377e36caf429a163721492f1b1c08deb9) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f14e3a605d4] , 2: (DeleteOp::_begin()+0x519) [0x55ce262f4099] 2020-11-18T18:42:47.730 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-18T12:02:21.315 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2020-11-18T12:02:20.101473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29907025880,"num_objects":10021,"num_object_clones":0,"num_object_copies":20042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9996,"num_whiteouts":0,"num_read":21560,"num_read_kb":111566,"num_write":52242,"num_write_kb":23304041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":25,"num_bytes_hit_set_archive":12609,"num_flush":3426,"num_flush_kb":10013824,"num_evict":3017,"num_evict_kb":8828532,"num_promote":3783,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47836,"ondisk_log_size":47836,"up":32,"acting":32,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":32,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":37893248,"kb_used_data":37592872,"kb_used_omap":0,"kb_used_meta":278144,"kb_avail":528337792,"statfs":{"total":579820584960,"available":541017899008,"internally_reserved":0,"allocated":38495100928,"data_stored":38345262544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":284819456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1626810387,"num_objects":537,"num_object_clones":0,"num_object_copies":1074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":537,"num_whiteouts":0,"num_read":1173,"num_read_kb":102087,"num_write":2624,"num_write_kb":930886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":63,"num_flush_kb":174864,"num_evict":65,"num_evict_kb":181271,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":-2,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"24","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130732+0000","last_change":"2020-11-18T11:59:16.220885+0000","last_active":"2020-11-18T12:01:41.130732+0000","last_peered":"2020-11-18T12:01:41.130732+0000","last_clean":"2020-11-18T12:01:41.130732+0000","last_became_active":"2020-11-18T11:59:16.220460+0000","last_became_peered":"2020-11-18T11:59:16.220460+0000","last_unstale":"2020-11-18T12:01:41.130732+0000","last_undegraded":"2020-11-18T12:01:41.130732+0000","last_fullsized":"2020-11-18T12:01:41.130732+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"23","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130021+0000","last_change":"2020-11-18T11:59:16.499247+0000","last_active":"2020-11-18T12:01:41.130021+0000","last_peered":"2020-11-18T12:01:41.130021+0000","last_clean":"2020-11-18T12:01:41.130021+0000","last_became_active":"2020-11-18T11:59:16.498717+0000","last_became_peered":"2020-11-18T11:59:16.498717+0000","last_unstale":"2020-11-18T12:01:41.130021+0000","last_undegraded":"2020-11-18T12:01:41.130021+0000","last_fullsized":"2020-11-18T12:01:41.130021+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"34'5488","reported_seq":"7600","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:05.424086+0000","last_change":"2020-11-18T11:59:27.193656+0000","last_active":"2020-11-18T12:02:05.424086+0000","last_peered":"2020-11-18T12:02:05.424086+0000","last_clean":"2020-11-18T12:02:05.424086+0000","last_became_active":"2020-11-18T11:59:27.193348+0000","last_became_peered":"2020-11-18T11:59:27.193348+0000","last_unstale":"2020-11-18T12:02:05.424086+0000","last_undegraded":"2020-11-18T12:02:05.424086+0000","last_fullsized":"2020-11-18T12:02:05.424086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5488,"ondisk_log_size":5488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":133229742,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":1879,"num_read_kb":944,"num_write":4937,"num_write_kb":1137382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":2934,"num_flush":871,"num_flush_kb":2574747,"num_evict":807,"num_evict_kb":2383573,"num_promote":973,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"34'6592","reported_seq":"8537","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.465679+0000","last_change":"2020-11-18T11:59:25.400522+0000","last_active":"2020-11-18T12:02:15.465679+0000","last_peered":"2020-11-18T12:02:15.465679+0000","last_clean":"2020-11-18T12:02:15.465679+0000","last_became_active":"2020-11-18T11:59:25.399994+0000","last_became_peered":"2020-11-18T11:59:25.399994+0000","last_unstale":"2020-11-18T12:02:15.465679+0000","last_undegraded":"2020-11-18T12:02:15.465679+0000","last_fullsized":"2020-11-18T12:02:15.465679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6592,"ondisk_log_size":6592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7292778019,"num_objects":2447,"num_object_clones":0,"num_object_copies":4894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2447,"num_whiteouts":0,"num_read":3524,"num_read_kb":41917,"num_write":8202,"num_write_kb":4621789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"24","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130666+0000","last_change":"2020-11-18T11:59:16.498778+0000","last_active":"2020-11-18T12:01:41.130666+0000","last_peered":"2020-11-18T12:01:41.130666+0000","last_clean":"2020-11-18T12:01:41.130666+0000","last_became_active":"2020-11-18T11:59:16.498407+0000","last_became_peered":"2020-11-18T11:59:16.498407+0000","last_unstale":"2020-11-18T12:01:41.130666+0000","last_undegraded":"2020-11-18T12:01:41.130666+0000","last_fullsized":"2020-11-18T12:01:41.130666+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"34'5230","reported_seq":"7236","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:06.598034+0000","last_change":"2020-11-18T11:59:27.469664+0000","last_active":"2020-11-18T12:02:06.598034+0000","last_peered":"2020-11-18T12:02:06.598034+0000","last_clean":"2020-11-18T12:02:06.598034+0000","last_became_active":"2020-11-18T11:59:27.469083+0000","last_became_peered":"2020-11-18T11:59:27.469083+0000","last_unstale":"2020-11-18T12:02:06.598034+0000","last_undegraded":"2020-11-18T12:02:06.598034+0000","last_fullsized":"2020-11-18T12:02:06.598034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5230,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":139283403,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":1837,"num_read_kb":905,"num_write":4791,"num_write_kb":1124747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":3402,"num_flush":843,"num_flush_kb":2441950,"num_evict":772,"num_evict_kb":2231779,"num_promote":919,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"34'6685","reported_seq":"8719","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.457184+0000","last_change":"2020-11-18T11:59:25.186658+0000","last_active":"2020-11-18T12:02:15.457184+0000","last_peered":"2020-11-18T12:02:15.457184+0000","last_clean":"2020-11-18T12:02:15.457184+0000","last_became_active":"2020-11-18T11:59:25.186414+0000","last_became_peered":"2020-11-18T11:59:25.186414+0000","last_unstale":"2020-11-18T12:02:15.457184+0000","last_undegraded":"2020-11-18T12:02:15.457184+0000","last_fullsized":"2020-11-18T12:02:15.457184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6685,"ondisk_log_size":6685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7453896300,"num_objects":2479,"num_object_clones":0,"num_object_copies":4958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2479,"num_whiteouts":0,"num_read":3574,"num_read_kb":19412,"num_write":8297,"num_write_kb":4811913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"23","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130006+0000","last_change":"2020-11-18T11:59:16.499200+0000","last_active":"2020-11-18T12:01:41.130006+0000","last_peered":"2020-11-18T12:01:41.130006+0000","last_clean":"2020-11-18T12:01:41.130006+0000","last_became_active":"2020-11-18T11:59:16.498419+0000","last_became_peered":"2020-11-18T11:59:16.498419+0000","last_unstale":"2020-11-18T12:01:41.130006+0000","last_undegraded":"2020-11-18T12:01:41.130006+0000","last_fullsized":"2020-11-18T12:01:41.130006+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"34'5183","reported_seq":"7371","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:06.410297+0000","last_change":"2020-11-18T11:59:27.469738+0000","last_active":"2020-11-18T12:02:06.410297+0000","last_peered":"2020-11-18T12:02:06.410297+0000","last_clean":"2020-11-18T12:02:06.410297+0000","last_became_active":"2020-11-18T11:59:27.469241+0000","last_became_peered":"2020-11-18T11:59:27.469241+0000","last_unstale":"2020-11-18T12:02:06.410297+0000","last_undegraded":"2020-11-18T12:02:06.410297+0000","last_fullsized":"2020-11-18T12:02:06.410297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5183,"ondisk_log_size":5183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":150727528,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":1927,"num_read_kb":886,"num_write":4708,"num_write_kb":1109010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":2455,"num_flush":821,"num_flush_kb":2396213,"num_evict":630,"num_evict_kb":1841924,"num_promote":896,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"34'6480","reported_seq":"8496","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.440474+0000","last_change":"2020-11-18T11:59:25.400594+0000","last_active":"2020-11-18T12:02:15.440474+0000","last_peered":"2020-11-18T12:02:15.440474+0000","last_clean":"2020-11-18T12:02:15.440474+0000","last_became_active":"2020-11-18T11:59:25.400132+0000","last_became_peered":"2020-11-18T11:59:25.400132+0000","last_unstale":"2020-11-18T12:02:15.440474+0000","last_undegraded":"2020-11-18T12:02:15.440474+0000","last_fullsized":"2020-11-18T12:02:15.440474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6480,"ondisk_log_size":6480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7283473264,"num_objects":2438,"num_object_clones":0,"num_object_copies":4876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":3498,"num_read_kb":27406,"num_write":8030,"num_write_kb":4749930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.129702+0000","last_change":"2020-11-18T11:59:16.499029+0000","last_active":"2020-11-18T12:01:41.129702+0000","last_peered":"2020-11-18T12:01:41.129702+0000","last_clean":"2020-11-18T12:01:41.129702+0000","last_became_active":"2020-11-18T11:59:16.498581+0000","last_became_peered":"2020-11-18T11:59:16.498581+0000","last_unstale":"2020-11-18T12:01:41.129702+0000","last_undegraded":"2020-11-18T12:01:41.129702+0000","last_fullsized":"2020-11-18T12:01:41.129702+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"34'5553","reported_seq":"7740","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:07.814557+0000","last_change":"2020-11-18T11:59:27.192743+0000","last_active":"2020-11-18T12:02:07.814557+0000","last_peered":"2020-11-18T12:02:07.814557+0000","last_clean":"2020-11-18T12:02:07.814557+0000","last_became_active":"2020-11-18T11:59:27.192298+0000","last_became_peered":"2020-11-18T11:59:27.192298+0000","last_unstale":"2020-11-18T12:02:07.814557+0000","last_undegraded":"2020-11-18T12:02:07.814557+0000","last_fullsized":"2020-11-18T12:02:07.814557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5553,"ondisk_log_size":5553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":139702080,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":1945,"num_read_kb":972,"num_write":5029,"num_write_kb":1151459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":3818,"num_flush":891,"num_flush_kb":2600914,"num_evict":808,"num_evict_kb":2371256,"num_promote":995,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"34'6625","reported_seq":"8418","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.441145+0000","last_change":"2020-11-18T11:59:25.400403+0000","last_active":"2020-11-18T12:02:15.441145+0000","last_peered":"2020-11-18T12:02:15.441145+0000","last_clean":"2020-11-18T12:02:15.441145+0000","last_became_active":"2020-11-18T11:59:25.400017+0000","last_became_peered":"2020-11-18T11:59:25.400017+0000","last_unstale":"2020-11-18T12:02:15.441145+0000","last_undegraded":"2020-11-18T12:02:15.441145+0000","last_fullsized":"2020-11-18T12:02:15.441145+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6625,"ondisk_log_size":6625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7313935544,"num_objects":2442,"num_object_clones":0,"num_object_copies":4884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2442,"num_whiteouts":0,"num_read":3376,"num_read_kb":19124,"num_write":8248,"num_write_kb":4597811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.132577+0000","last_change":"2020-11-18T11:59:16.174660+0000","last_active":"2020-11-18T12:01:41.132577+0000","last_peered":"2020-11-18T12:01:41.132577+0000","last_clean":"2020-11-18T12:01:41.132577+0000","last_became_active":"2020-11-18T11:59:16.174201+0000","last_became_peered":"2020-11-18T11:59:16.174201+0000","last_unstale":"2020-11-18T12:01:41.132577+0000","last_undegraded":"2020-11-18T12:01:41.132577+0000","last_fullsized":"2020-11-18T12:01:41.132577+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.132535+0000","last_change":"2020-11-18T11:59:16.174652+0000","last_active":"2020-11-18T12:01:41.132535+0000","last_peered":"2020-11-18T12:01:41.132535+0000","last_clean":"2020-11-18T12:01:41.132535+0000","last_became_active":"2020-11-18T11:59:16.174176+0000","last_became_peered":"2020-11-18T11:59:16.174176+0000","last_unstale":"2020-11-18T12:01:41.132535+0000","last_undegraded":"2020-11-18T12:01:41.132535+0000","last_fullsized":"2020-11-18T12:01:41.132535+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.129724+0000","last_change":"2020-11-18T11:59:16.220833+0000","last_active":"2020-11-18T12:01:41.129724+0000","last_peered":"2020-11-18T12:01:41.129724+0000","last_clean":"2020-11-18T12:01:41.129724+0000","last_became_active":"2020-11-18T11:59:16.220457+0000","last_became_peered":"2020-11-18T11:59:16.220457+0000","last_unstale":"2020-11-18T12:01:41.129724+0000","last_undegraded":"2020-11-18T12:01:41.129724+0000","last_fullsized":"2020-11-18T12:01:41.129724+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":562942753,"num_objects":215,"num_object_clones":0,"num_object_copies":430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":190,"num_whiteouts":0,"num_read":7588,"num_read_kb":3707,"num_write":19465,"num_write_kb":4522598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":25,"num_bytes_hit_set_archive":12609,"num_flush":3426,"num_flush_kb":10013824,"num_evict":3017,"num_evict_kb":8828532,"num_promote":3783,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":509820928,"data_stored":505904942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21454,"ondisk_log_size":21454,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":4,"stat_sum":{"num_bytes":29344083127,"num_objects":9806,"num_object_clones":0,"num_object_copies":19612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9806,"num_whiteouts":0,"num_read":13972,"num_read_kb":107859,"num_write":32777,"num_write_kb":18781443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":37982961664,"data_stored":37838060392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26382,"ondisk_log_size":26382,"up":8,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640295,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9432696,"kb_used_data":9372000,"kb_used_omap":0,"kb_used_meta":51776,"kb_avail":84939144,"statfs":{"total":96636764160,"available":86977683456,"internally_reserved":0,"allocated":9596928000,"data_stored":9559886176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53018624},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.66200000000000003,"15min":0.64900000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60399999999999998,"15min":0.59399999999999997},"min":{"1min":0.41999999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82499999999999996,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.69699999999999995}]},{"osd":1,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67600000000000005,"15min":0.68200000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.77000000000000002,"15min":0.78300000000000003},"min":{"1min":0.51200000000000001,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.202,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.872}]},{"osd":2,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.80600000000000005,"15min":0.79000000000000004},"min":{"1min":0.52000000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.7010000000000001,"5min":1.7010000000000001,"15min":1.7010000000000001},"last":1.1679999999999999},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.86499999999999999,"15min":0.876},"min":{"1min":0.57099999999999995,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":1.026}]},{"osd":3,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.73499999999999999,"15min":0.73299999999999998},"min":{"1min":0.45100000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.80500000000000005,"15min":0.80900000000000005},"min":{"1min":0.56999999999999995,"5min":0.439,"15min":0.439},"max":{"1min":0.98399999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.98399999999999999}]},{"osd":4,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72299999999999998,"15min":0.74299999999999999},"min":{"1min":0.44800000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.875,"15min":0.876},"min":{"1min":0.64400000000000002,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.504,"5min":1.504,"15min":1.504},"last":1.1950000000000001}]}]},{"osd":4,"up_from":12,"seq":51539607592,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155880,"kb_used_data":119456,"kb_used_omap":0,"kb_used_meta":36416,"kb_avail":94215960,"statfs":{"total":96636764160,"available":96477143040,"internally_reserved":0,"allocated":122322944,"data_stored":121186256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37289984},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.68500000000000005,"15min":0.70599999999999996},"min":{"1min":0.40000000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.81499999999999995,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.64100000000000001,"15min":0.65200000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.878,"5min":1.131,"15min":1.131},"last":0.65300000000000002}]},{"osd":1,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.70099999999999996,"15min":0.70999999999999996},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.97499999999999998,"5min":1.145,"15min":1.145},"last":0.746},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.75700000000000001,"15min":0.77300000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.95599999999999996,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.68700000000000006}]},{"osd":2,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.81999999999999995,"15min":0.83799999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.02,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.84499999999999997,"15min":0.86099999999999999},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.054,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.879}]},{"osd":3,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70099999999999996,"15min":0.70999999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.78500000000000003,"15min":0.78900000000000003},"min":{"1min":0.57999999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.93600000000000005,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.94999999999999996}]},{"osd":5,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72799999999999998,"15min":0.72899999999999998},"min":{"1min":0.57599999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.873,"15min":0.88300000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.0349999999999999,"5min":1.264,"15min":1.264},"last":1.0209999999999999}]}]},{"osd":3,"up_from":12,"seq":51539607591,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4595176,"kb_used_data":4564824,"kb_used_omap":0,"kb_used_meta":28864,"kb_avail":89776664,"statfs":{"total":96636764160,"available":91931303936,"internally_reserved":0,"allocated":4674379776,"data_stored":4655592721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29556736},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.59699999999999998,"15min":0.58699999999999997},"min":{"1min":0.38400000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.421,"5min":1.421,"15min":1.421},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63,"15min":0.63500000000000001},"min":{"1min":0.45000000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91700000000000004,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.89200000000000002}]},{"osd":1,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.68700000000000006,"15min":0.68400000000000005},"min":{"1min":0.496,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.51700000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.7030000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":0.85599999999999998}]},{"osd":2,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.503,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.81399999999999995,"15min":0.83499999999999996},"min":{"1min":0.56599999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":1.0209999999999999}]},{"osd":4,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.46100000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.747},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.78000000000000003,"15min":0.79800000000000004},"min":{"1min":0.49099999999999999,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.046,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.93100000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.82899999999999996,"15min":0.80300000000000005},"min":{"1min":0.56399999999999995,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.137},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.83899999999999997,"15min":0.81499999999999995},"min":{"1min":0.65400000000000003,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.3199999999999998,"5min":2.3199999999999998,"15min":2.3199999999999998},"last":0.997}]}]},{"osd":2,"up_from":12,"seq":51539607591,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9531084,"kb_used_data":9470512,"kb_used_omap":0,"kb_used_meta":51328,"kb_avail":84840756,"statfs":{"total":96636764160,"available":86876934144,"internally_reserved":0,"allocated":9697804288,"data_stored":9661420712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52559872},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.502,"15min":0.5},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.438},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.62,"15min":0.622},"min":{"1min":0.40899999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93799999999999994,"5min":1.464,"15min":1.464},"last":0.48799999999999999}]},{"osd":1,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.68999999999999995,"15min":0.68899999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56799999999999995,"15min":0.57399999999999995},"min":{"1min":0.44500000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.67800000000000005,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.53300000000000003}]},{"osd":3,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.69299999999999995,"15min":0.69799999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0069999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.77000000000000002,"15min":0.78000000000000003},"min":{"1min":0.54300000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.026,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.79200000000000004}]},{"osd":4,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.89400000000000002,"15min":0.89800000000000002},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.101,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.876,"15min":0.88600000000000001},"min":{"1min":0.58899999999999997,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.294,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.75700000000000001}]},{"osd":5,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":2.2210000000000001,"15min":2.4700000000000002},"min":{"1min":0.60699999999999998,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.212,"5min":34.686999999999998,"15min":34.686999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.79800000000000004,"5min":2.266,"15min":2.5110000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.0029999999999999,"5min":34.798000000000002,"15min":34.798000000000002},"last":0.71999999999999997}]}]},{"osd":0,"up_from":12,"seq":51539607591,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155880,"kb_used_data":119456,"kb_used_omap":0,"kb_used_meta":36416,"kb_avail":94215960,"statfs":{"total":96636764160,"available":96477143040,"internally_reserved":0,"allocated":122322944,"data_stored":121186256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37289984},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.64000000000000001,"15min":0.65900000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0529999999999999,"5min":1.754,"15min":1.754},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.57999999999999996,"15min":0.56799999999999995},"min":{"1min":0.41999999999999998,"5min":0.37,"15min":0.37},"max":{"1min":1.639,"5min":1.639,"15min":1.639},"last":0.64100000000000001}]},{"osd":2,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.71499999999999997,"15min":0.72199999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.995,"5min":2.4079999999999999,"15min":2.4079999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.65200000000000002,"15min":0.64700000000000002},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.504}]},{"osd":3,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.76600000000000001,"15min":0.77400000000000002},"min":{"1min":0.51800000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1899999999999999,"5min":1.218,"15min":1.218},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.77500000000000002,"15min":0.79200000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.93600000000000005,"5min":1.181,"15min":1.181},"last":0.60099999999999998}]},{"osd":4,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.89400000000000002,"15min":0.91000000000000003},"min":{"1min":0.56799999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.069,"5min":1.9790000000000001,"15min":1.9790000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.91500000000000004,"15min":0.92700000000000005},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.3200000000000001,"5min":2.141,"15min":2.141},"last":0.92700000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.88700000000000001,"15min":0.88800000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.98699999999999999,"15min":1.002},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.276,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":0.79900000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640296,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14022532,"kb_used_data":13946624,"kb_used_omap":0,"kb_used_meta":73344,"kb_avail":80349308,"statfs":{"total":96636764160,"available":82277691392,"internally_reserved":0,"allocated":14281342976,"data_stored":14225990423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":75104256},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.58699999999999997,"15min":0.58099999999999996},"min":{"1min":0.40799999999999997,"5min":0.373,"15min":0.373},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.56899999999999995,"15min":0.55900000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.68400000000000005}]},{"osd":2,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57799999999999996,"15min":0.57699999999999996},"min":{"1min":0.41599999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.746,"5min":0.70199999999999996,"15min":0.69499999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.115,"5min":1.179,"15min":1.179},"last":0.61499999999999999}]},{"osd":3,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.753,"15min":0.76200000000000001},"min":{"1min":0.46300000000000002,"5min":0.441,"15min":0.441},"max":{"1min":1.1040000000000001,"5min":1.139,"15min":1.139},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.80000000000000004,"15min":0.80200000000000005},"min":{"1min":0.48099999999999998,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.96999999999999997}]},{"osd":4,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.90500000000000003,"15min":0.91700000000000004},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.0700000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":1.117},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.88900000000000001,"15min":0.89800000000000002},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.244,"5min":1.536,"15min":1.536},"last":0.93200000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:02:13 2020","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.93700000000000006,"15min":0.93200000000000005},"min":{"1min":0.70199999999999996,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.387,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.0249999999999999,"5min":0.95299999999999996,"15min":0.94099999999999995},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.48,"5min":1.48,"15min":1.48},"last":1.077}]}]}]}} , 2020-11-18T12:02:21.319 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-11-18T11:59:15.163082+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=11, tm_min=59, tm_sec=15, tm_wday=2, tm_yday=323, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=12, tm_min=2, tm_sec=16, tm_wday=2, tm_yday=323, tm_isdst=0) , 2020-11-18T12:02:21.319 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-11-18T11:59:15.163082+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=11, tm_min=59, tm_sec=15, tm_wday=2, tm_yday=323, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=12, tm_min=2, tm_sec=16, tm_wday=2, tm_yday=323, tm_isdst=0) 2020-11-18T18:42:47.732 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-18T12:03:26.330 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi045.front.sepia.ceph.com , 2020-11-18T12:03:26.330 INFO:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 , 2020-11-18T12:03:26.411 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi072.front.sepia.ceph.com 2020-11-18T18:42:47.732 WARNING:scrape:Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' , 2020-11-18T12:05:24.383 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2020-11-18T18:42:47.733 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status 2020-11-18T18:42:47.837 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-7273-gfbf83bb3 (fbf83bb377e36caf429a163721492f1b1c08deb9) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f14e3a605d4] , 2: (DeleteOp::_begin()+0x519) [0x55ce262f4099] 2020-11-18T18:42:47.839 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-18T12:02:21.315 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2020-11-18T12:02:20.101473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29907025880,"num_objects":10021,"num_object_clones":0,"num_object_copies":20042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9996,"num_whiteouts":0,"num_read":21560,"num_read_kb":111566,"num_write":52242,"num_write_kb":23304041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":25,"num_bytes_hit_set_archive":12609,"num_flush":3426,"num_flush_kb":10013824,"num_evict":3017,"num_evict_kb":8828532,"num_promote":3783,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47836,"ondisk_log_size":47836,"up":32,"acting":32,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":32,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":37893248,"kb_used_data":37592872,"kb_used_omap":0,"kb_used_meta":278144,"kb_avail":528337792,"statfs":{"total":579820584960,"available":541017899008,"internally_reserved":0,"allocated":38495100928,"data_stored":38345262544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":284819456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1626810387,"num_objects":537,"num_object_clones":0,"num_object_copies":1074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":537,"num_whiteouts":0,"num_read":1173,"num_read_kb":102087,"num_write":2624,"num_write_kb":930886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":63,"num_flush_kb":174864,"num_evict":65,"num_evict_kb":181271,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":-2,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"24","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130732+0000","last_change":"2020-11-18T11:59:16.220885+0000","last_active":"2020-11-18T12:01:41.130732+0000","last_peered":"2020-11-18T12:01:41.130732+0000","last_clean":"2020-11-18T12:01:41.130732+0000","last_became_active":"2020-11-18T11:59:16.220460+0000","last_became_peered":"2020-11-18T11:59:16.220460+0000","last_unstale":"2020-11-18T12:01:41.130732+0000","last_undegraded":"2020-11-18T12:01:41.130732+0000","last_fullsized":"2020-11-18T12:01:41.130732+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"23","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130021+0000","last_change":"2020-11-18T11:59:16.499247+0000","last_active":"2020-11-18T12:01:41.130021+0000","last_peered":"2020-11-18T12:01:41.130021+0000","last_clean":"2020-11-18T12:01:41.130021+0000","last_became_active":"2020-11-18T11:59:16.498717+0000","last_became_peered":"2020-11-18T11:59:16.498717+0000","last_unstale":"2020-11-18T12:01:41.130021+0000","last_undegraded":"2020-11-18T12:01:41.130021+0000","last_fullsized":"2020-11-18T12:01:41.130021+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"34'5488","reported_seq":"7600","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:05.424086+0000","last_change":"2020-11-18T11:59:27.193656+0000","last_active":"2020-11-18T12:02:05.424086+0000","last_peered":"2020-11-18T12:02:05.424086+0000","last_clean":"2020-11-18T12:02:05.424086+0000","last_became_active":"2020-11-18T11:59:27.193348+0000","last_became_peered":"2020-11-18T11:59:27.193348+0000","last_unstale":"2020-11-18T12:02:05.424086+0000","last_undegraded":"2020-11-18T12:02:05.424086+0000","last_fullsized":"2020-11-18T12:02:05.424086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5488,"ondisk_log_size":5488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":133229742,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":1879,"num_read_kb":944,"num_write":4937,"num_write_kb":1137382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":2934,"num_flush":871,"num_flush_kb":2574747,"num_evict":807,"num_evict_kb":2383573,"num_promote":973,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"34'6592","reported_seq":"8537","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.465679+0000","last_change":"2020-11-18T11:59:25.400522+0000","last_active":"2020-11-18T12:02:15.465679+0000","last_peered":"2020-11-18T12:02:15.465679+0000","last_clean":"2020-11-18T12:02:15.465679+0000","last_became_active":"2020-11-18T11:59:25.399994+0000","last_became_peered":"2020-11-18T11:59:25.399994+0000","last_unstale":"2020-11-18T12:02:15.465679+0000","last_undegraded":"2020-11-18T12:02:15.465679+0000","last_fullsized":"2020-11-18T12:02:15.465679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6592,"ondisk_log_size":6592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7292778019,"num_objects":2447,"num_object_clones":0,"num_object_copies":4894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2447,"num_whiteouts":0,"num_read":3524,"num_read_kb":41917,"num_write":8202,"num_write_kb":4621789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"24","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130666+0000","last_change":"2020-11-18T11:59:16.498778+0000","last_active":"2020-11-18T12:01:41.130666+0000","last_peered":"2020-11-18T12:01:41.130666+0000","last_clean":"2020-11-18T12:01:41.130666+0000","last_became_active":"2020-11-18T11:59:16.498407+0000","last_became_peered":"2020-11-18T11:59:16.498407+0000","last_unstale":"2020-11-18T12:01:41.130666+0000","last_undegraded":"2020-11-18T12:01:41.130666+0000","last_fullsized":"2020-11-18T12:01:41.130666+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"34'5230","reported_seq":"7236","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:06.598034+0000","last_change":"2020-11-18T11:59:27.469664+0000","last_active":"2020-11-18T12:02:06.598034+0000","last_peered":"2020-11-18T12:02:06.598034+0000","last_clean":"2020-11-18T12:02:06.598034+0000","last_became_active":"2020-11-18T11:59:27.469083+0000","last_became_peered":"2020-11-18T11:59:27.469083+0000","last_unstale":"2020-11-18T12:02:06.598034+0000","last_undegraded":"2020-11-18T12:02:06.598034+0000","last_fullsized":"2020-11-18T12:02:06.598034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5230,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":139283403,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":1837,"num_read_kb":905,"num_write":4791,"num_write_kb":1124747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":3402,"num_flush":843,"num_flush_kb":2441950,"num_evict":772,"num_evict_kb":2231779,"num_promote":919,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"34'6685","reported_seq":"8719","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.457184+0000","last_change":"2020-11-18T11:59:25.186658+0000","last_active":"2020-11-18T12:02:15.457184+0000","last_peered":"2020-11-18T12:02:15.457184+0000","last_clean":"2020-11-18T12:02:15.457184+0000","last_became_active":"2020-11-18T11:59:25.186414+0000","last_became_peered":"2020-11-18T11:59:25.186414+0000","last_unstale":"2020-11-18T12:02:15.457184+0000","last_undegraded":"2020-11-18T12:02:15.457184+0000","last_fullsized":"2020-11-18T12:02:15.457184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6685,"ondisk_log_size":6685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7453896300,"num_objects":2479,"num_object_clones":0,"num_object_copies":4958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2479,"num_whiteouts":0,"num_read":3574,"num_read_kb":19412,"num_write":8297,"num_write_kb":4811913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"23","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130006+0000","last_change":"2020-11-18T11:59:16.499200+0000","last_active":"2020-11-18T12:01:41.130006+0000","last_peered":"2020-11-18T12:01:41.130006+0000","last_clean":"2020-11-18T12:01:41.130006+0000","last_became_active":"2020-11-18T11:59:16.498419+0000","last_became_peered":"2020-11-18T11:59:16.498419+0000","last_unstale":"2020-11-18T12:01:41.130006+0000","last_undegraded":"2020-11-18T12:01:41.130006+0000","last_fullsized":"2020-11-18T12:01:41.130006+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"34'5183","reported_seq":"7371","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:06.410297+0000","last_change":"2020-11-18T11:59:27.469738+0000","last_active":"2020-11-18T12:02:06.410297+0000","last_peered":"2020-11-18T12:02:06.410297+0000","last_clean":"2020-11-18T12:02:06.410297+0000","last_became_active":"2020-11-18T11:59:27.469241+0000","last_became_peered":"2020-11-18T11:59:27.469241+0000","last_unstale":"2020-11-18T12:02:06.410297+0000","last_undegraded":"2020-11-18T12:02:06.410297+0000","last_fullsized":"2020-11-18T12:02:06.410297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5183,"ondisk_log_size":5183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":150727528,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":1927,"num_read_kb":886,"num_write":4708,"num_write_kb":1109010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":2455,"num_flush":821,"num_flush_kb":2396213,"num_evict":630,"num_evict_kb":1841924,"num_promote":896,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"34'6480","reported_seq":"8496","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.440474+0000","last_change":"2020-11-18T11:59:25.400594+0000","last_active":"2020-11-18T12:02:15.440474+0000","last_peered":"2020-11-18T12:02:15.440474+0000","last_clean":"2020-11-18T12:02:15.440474+0000","last_became_active":"2020-11-18T11:59:25.400132+0000","last_became_peered":"2020-11-18T11:59:25.400132+0000","last_unstale":"2020-11-18T12:02:15.440474+0000","last_undegraded":"2020-11-18T12:02:15.440474+0000","last_fullsized":"2020-11-18T12:02:15.440474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6480,"ondisk_log_size":6480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7283473264,"num_objects":2438,"num_object_clones":0,"num_object_copies":4876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":3498,"num_read_kb":27406,"num_write":8030,"num_write_kb":4749930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.129702+0000","last_change":"2020-11-18T11:59:16.499029+0000","last_active":"2020-11-18T12:01:41.129702+0000","last_peered":"2020-11-18T12:01:41.129702+0000","last_clean":"2020-11-18T12:01:41.129702+0000","last_became_active":"2020-11-18T11:59:16.498581+0000","last_became_peered":"2020-11-18T11:59:16.498581+0000","last_unstale":"2020-11-18T12:01:41.129702+0000","last_undegraded":"2020-11-18T12:01:41.129702+0000","last_fullsized":"2020-11-18T12:01:41.129702+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"34'5553","reported_seq":"7740","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:07.814557+0000","last_change":"2020-11-18T11:59:27.192743+0000","last_active":"2020-11-18T12:02:07.814557+0000","last_peered":"2020-11-18T12:02:07.814557+0000","last_clean":"2020-11-18T12:02:07.814557+0000","last_became_active":"2020-11-18T11:59:27.192298+0000","last_became_peered":"2020-11-18T11:59:27.192298+0000","last_unstale":"2020-11-18T12:02:07.814557+0000","last_undegraded":"2020-11-18T12:02:07.814557+0000","last_fullsized":"2020-11-18T12:02:07.814557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5553,"ondisk_log_size":5553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":139702080,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":1945,"num_read_kb":972,"num_write":5029,"num_write_kb":1151459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":3818,"num_flush":891,"num_flush_kb":2600914,"num_evict":808,"num_evict_kb":2371256,"num_promote":995,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"34'6625","reported_seq":"8418","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.441145+0000","last_change":"2020-11-18T11:59:25.400403+0000","last_active":"2020-11-18T12:02:15.441145+0000","last_peered":"2020-11-18T12:02:15.441145+0000","last_clean":"2020-11-18T12:02:15.441145+0000","last_became_active":"2020-11-18T11:59:25.400017+0000","last_became_peered":"2020-11-18T11:59:25.400017+0000","last_unstale":"2020-11-18T12:02:15.441145+0000","last_undegraded":"2020-11-18T12:02:15.441145+0000","last_fullsized":"2020-11-18T12:02:15.441145+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6625,"ondisk_log_size":6625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7313935544,"num_objects":2442,"num_object_clones":0,"num_object_copies":4884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2442,"num_whiteouts":0,"num_read":3376,"num_read_kb":19124,"num_write":8248,"num_write_kb":4597811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.132577+0000","last_change":"2020-11-18T11:59:16.174660+0000","last_active":"2020-11-18T12:01:41.132577+0000","last_peered":"2020-11-18T12:01:41.132577+0000","last_clean":"2020-11-18T12:01:41.132577+0000","last_became_active":"2020-11-18T11:59:16.174201+0000","last_became_peered":"2020-11-18T11:59:16.174201+0000","last_unstale":"2020-11-18T12:01:41.132577+0000","last_undegraded":"2020-11-18T12:01:41.132577+0000","last_fullsized":"2020-11-18T12:01:41.132577+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.132535+0000","last_change":"2020-11-18T11:59:16.174652+0000","last_active":"2020-11-18T12:01:41.132535+0000","last_peered":"2020-11-18T12:01:41.132535+0000","last_clean":"2020-11-18T12:01:41.132535+0000","last_became_active":"2020-11-18T11:59:16.174176+0000","last_became_peered":"2020-11-18T11:59:16.174176+0000","last_unstale":"2020-11-18T12:01:41.132535+0000","last_undegraded":"2020-11-18T12:01:41.132535+0000","last_fullsized":"2020-11-18T12:01:41.132535+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.129724+0000","last_change":"2020-11-18T11:59:16.220833+0000","last_active":"2020-11-18T12:01:41.129724+0000","last_peered":"2020-11-18T12:01:41.129724+0000","last_clean":"2020-11-18T12:01:41.129724+0000","last_became_active":"2020-11-18T11:59:16.220457+0000","last_became_peered":"2020-11-18T11:59:16.220457+0000","last_unstale":"2020-11-18T12:01:41.129724+0000","last_undegraded":"2020-11-18T12:01:41.129724+0000","last_fullsized":"2020-11-18T12:01:41.129724+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":562942753,"num_objects":215,"num_object_clones":0,"num_object_copies":430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":190,"num_whiteouts":0,"num_read":7588,"num_read_kb":3707,"num_write":19465,"num_write_kb":4522598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":25,"num_bytes_hit_set_archive":12609,"num_flush":3426,"num_flush_kb":10013824,"num_evict":3017,"num_evict_kb":8828532,"num_promote":3783,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":509820928,"data_stored":505904942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21454,"ondisk_log_size":21454,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":4,"stat_sum":{"num_bytes":29344083127,"num_objects":9806,"num_object_clones":0,"num_object_copies":19612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9806,"num_whiteouts":0,"num_read":13972,"num_read_kb":107859,"num_write":32777,"num_write_kb":18781443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":37982961664,"data_stored":37838060392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26382,"ondisk_log_size":26382,"up":8,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640295,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9432696,"kb_used_data":9372000,"kb_used_omap":0,"kb_used_meta":51776,"kb_avail":84939144,"statfs":{"total":96636764160,"available":86977683456,"internally_reserved":0,"allocated":9596928000,"data_stored":9559886176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53018624},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.66200000000000003,"15min":0.64900000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60399999999999998,"15min":0.59399999999999997},"min":{"1min":0.41999999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82499999999999996,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.69699999999999995}]},{"osd":1,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67600000000000005,"15min":0.68200000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.77000000000000002,"15min":0.78300000000000003},"min":{"1min":0.51200000000000001,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.202,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.872}]},{"osd":2,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.80600000000000005,"15min":0.79000000000000004},"min":{"1min":0.52000000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.7010000000000001,"5min":1.7010000000000001,"15min":1.7010000000000001},"last":1.1679999999999999},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.86499999999999999,"15min":0.876},"min":{"1min":0.57099999999999995,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":1.026}]},{"osd":3,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.73499999999999999,"15min":0.73299999999999998},"min":{"1min":0.45100000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.80500000000000005,"15min":0.80900000000000005},"min":{"1min":0.56999999999999995,"5min":0.439,"15min":0.439},"max":{"1min":0.98399999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.98399999999999999}]},{"osd":4,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72299999999999998,"15min":0.74299999999999999},"min":{"1min":0.44800000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.875,"15min":0.876},"min":{"1min":0.64400000000000002,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.504,"5min":1.504,"15min":1.504},"last":1.1950000000000001}]}]},{"osd":4,"up_from":12,"seq":51539607592,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155880,"kb_used_data":119456,"kb_used_omap":0,"kb_used_meta":36416,"kb_avail":94215960,"statfs":{"total":96636764160,"available":96477143040,"internally_reserved":0,"allocated":122322944,"data_stored":121186256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37289984},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.68500000000000005,"15min":0.70599999999999996},"min":{"1min":0.40000000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.81499999999999995,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.64100000000000001,"15min":0.65200000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.878,"5min":1.131,"15min":1.131},"last":0.65300000000000002}]},{"osd":1,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.70099999999999996,"15min":0.70999999999999996},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.97499999999999998,"5min":1.145,"15min":1.145},"last":0.746},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.75700000000000001,"15min":0.77300000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.95599999999999996,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.68700000000000006}]},{"osd":2,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.81999999999999995,"15min":0.83799999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.02,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.84499999999999997,"15min":0.86099999999999999},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.054,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.879}]},{"osd":3,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70099999999999996,"15min":0.70999999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.78500000000000003,"15min":0.78900000000000003},"min":{"1min":0.57999999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.93600000000000005,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.94999999999999996}]},{"osd":5,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72799999999999998,"15min":0.72899999999999998},"min":{"1min":0.57599999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.873,"15min":0.88300000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.0349999999999999,"5min":1.264,"15min":1.264},"last":1.0209999999999999}]}]},{"osd":3,"up_from":12,"seq":51539607591,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4595176,"kb_used_data":4564824,"kb_used_omap":0,"kb_used_meta":28864,"kb_avail":89776664,"statfs":{"total":96636764160,"available":91931303936,"internally_reserved":0,"allocated":4674379776,"data_stored":4655592721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29556736},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.59699999999999998,"15min":0.58699999999999997},"min":{"1min":0.38400000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.421,"5min":1.421,"15min":1.421},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63,"15min":0.63500000000000001},"min":{"1min":0.45000000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91700000000000004,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.89200000000000002}]},{"osd":1,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.68700000000000006,"15min":0.68400000000000005},"min":{"1min":0.496,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.51700000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.7030000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":0.85599999999999998}]},{"osd":2,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.503,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.81399999999999995,"15min":0.83499999999999996},"min":{"1min":0.56599999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":1.0209999999999999}]},{"osd":4,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.46100000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.747},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.78000000000000003,"15min":0.79800000000000004},"min":{"1min":0.49099999999999999,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.046,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.93100000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.82899999999999996,"15min":0.80300000000000005},"min":{"1min":0.56399999999999995,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.137},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.83899999999999997,"15min":0.81499999999999995},"min":{"1min":0.65400000000000003,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.3199999999999998,"5min":2.3199999999999998,"15min":2.3199999999999998},"last":0.997}]}]},{"osd":2,"up_from":12,"seq":51539607591,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9531084,"kb_used_data":9470512,"kb_used_omap":0,"kb_used_meta":51328,"kb_avail":84840756,"statfs":{"total":96636764160,"available":86876934144,"internally_reserved":0,"allocated":9697804288,"data_stored":9661420712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52559872},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.502,"15min":0.5},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.438},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.62,"15min":0.622},"min":{"1min":0.40899999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93799999999999994,"5min":1.464,"15min":1.464},"last":0.48799999999999999}]},{"osd":1,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.68999999999999995,"15min":0.68899999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56799999999999995,"15min":0.57399999999999995},"min":{"1min":0.44500000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.67800000000000005,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.53300000000000003}]},{"osd":3,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.69299999999999995,"15min":0.69799999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0069999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.77000000000000002,"15min":0.78000000000000003},"min":{"1min":0.54300000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.026,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.79200000000000004}]},{"osd":4,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.89400000000000002,"15min":0.89800000000000002},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.101,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.876,"15min":0.88600000000000001},"min":{"1min":0.58899999999999997,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.294,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.75700000000000001}]},{"osd":5,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":2.2210000000000001,"15min":2.4700000000000002},"min":{"1min":0.60699999999999998,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.212,"5min":34.686999999999998,"15min":34.686999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.79800000000000004,"5min":2.266,"15min":2.5110000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.0029999999999999,"5min":34.798000000000002,"15min":34.798000000000002},"last":0.71999999999999997}]}]},{"osd":0,"up_from":12,"seq":51539607591,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155880,"kb_used_data":119456,"kb_used_omap":0,"kb_used_meta":36416,"kb_avail":94215960,"statfs":{"total":96636764160,"available":96477143040,"internally_reserved":0,"allocated":122322944,"data_stored":121186256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37289984},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.64000000000000001,"15min":0.65900000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0529999999999999,"5min":1.754,"15min":1.754},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.57999999999999996,"15min":0.56799999999999995},"min":{"1min":0.41999999999999998,"5min":0.37,"15min":0.37},"max":{"1min":1.639,"5min":1.639,"15min":1.639},"last":0.64100000000000001}]},{"osd":2,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.71499999999999997,"15min":0.72199999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.995,"5min":2.4079999999999999,"15min":2.4079999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.65200000000000002,"15min":0.64700000000000002},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.504}]},{"osd":3,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.76600000000000001,"15min":0.77400000000000002},"min":{"1min":0.51800000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1899999999999999,"5min":1.218,"15min":1.218},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.77500000000000002,"15min":0.79200000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.93600000000000005,"5min":1.181,"15min":1.181},"last":0.60099999999999998}]},{"osd":4,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.89400000000000002,"15min":0.91000000000000003},"min":{"1min":0.56799999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.069,"5min":1.9790000000000001,"15min":1.9790000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.91500000000000004,"15min":0.92700000000000005},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.3200000000000001,"5min":2.141,"15min":2.141},"last":0.92700000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.88700000000000001,"15min":0.88800000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.98699999999999999,"15min":1.002},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.276,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":0.79900000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640296,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14022532,"kb_used_data":13946624,"kb_used_omap":0,"kb_used_meta":73344,"kb_avail":80349308,"statfs":{"total":96636764160,"available":82277691392,"internally_reserved":0,"allocated":14281342976,"data_stored":14225990423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":75104256},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.58699999999999997,"15min":0.58099999999999996},"min":{"1min":0.40799999999999997,"5min":0.373,"15min":0.373},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.56899999999999995,"15min":0.55900000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.68400000000000005}]},{"osd":2,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57799999999999996,"15min":0.57699999999999996},"min":{"1min":0.41599999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.746,"5min":0.70199999999999996,"15min":0.69499999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.115,"5min":1.179,"15min":1.179},"last":0.61499999999999999}]},{"osd":3,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.753,"15min":0.76200000000000001},"min":{"1min":0.46300000000000002,"5min":0.441,"15min":0.441},"max":{"1min":1.1040000000000001,"5min":1.139,"15min":1.139},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.80000000000000004,"15min":0.80200000000000005},"min":{"1min":0.48099999999999998,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.96999999999999997}]},{"osd":4,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.90500000000000003,"15min":0.91700000000000004},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.0700000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":1.117},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.88900000000000001,"15min":0.89800000000000002},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.244,"5min":1.536,"15min":1.536},"last":0.93200000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:02:13 2020","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.93700000000000006,"15min":0.93200000000000005},"min":{"1min":0.70199999999999996,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.387,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.0249999999999999,"5min":0.95299999999999996,"15min":0.94099999999999995},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.48,"5min":1.48,"15min":1.48},"last":1.077}]}]}]}} , 2020-11-18T12:02:21.319 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-11-18T11:59:15.163082+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=11, tm_min=59, tm_sec=15, tm_wday=2, tm_yday=323, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=12, tm_min=2, tm_sec=16, tm_wday=2, tm_yday=323, tm_isdst=0) , 2020-11-18T12:02:21.319 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-11-18T11:59:15.163082+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=11, tm_min=59, tm_sec=15, tm_wday=2, tm_yday=323, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=12, tm_min=2, tm_sec=16, tm_wday=2, tm_yday=323, tm_isdst=0) 2020-11-18T18:42:47.840 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-18T12:03:26.330 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi045.front.sepia.ceph.com , 2020-11-18T12:03:26.330 INFO:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 , 2020-11-18T12:03:26.411 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi072.front.sepia.ceph.com 2020-11-18T18:42:47.841 WARNING:scrape:Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' , 2020-11-18T12:05:24.383 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2020-11-18T18:42:47.842 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status 2020-11-18T18:42:47.950 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-7273-gfbf83bb3 (fbf83bb377e36caf429a163721492f1b1c08deb9) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f14e3a605d4] , 2: (DeleteOp::_begin()+0x519) [0x55ce262f4099] 2020-11-18T18:42:47.952 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-18T12:02:21.315 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2020-11-18T12:02:20.101473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29907025880,"num_objects":10021,"num_object_clones":0,"num_object_copies":20042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9996,"num_whiteouts":0,"num_read":21560,"num_read_kb":111566,"num_write":52242,"num_write_kb":23304041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":25,"num_bytes_hit_set_archive":12609,"num_flush":3426,"num_flush_kb":10013824,"num_evict":3017,"num_evict_kb":8828532,"num_promote":3783,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47836,"ondisk_log_size":47836,"up":32,"acting":32,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":32,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":37893248,"kb_used_data":37592872,"kb_used_omap":0,"kb_used_meta":278144,"kb_avail":528337792,"statfs":{"total":579820584960,"available":541017899008,"internally_reserved":0,"allocated":38495100928,"data_stored":38345262544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":284819456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1626810387,"num_objects":537,"num_object_clones":0,"num_object_copies":1074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":537,"num_whiteouts":0,"num_read":1173,"num_read_kb":102087,"num_write":2624,"num_write_kb":930886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":63,"num_flush_kb":174864,"num_evict":65,"num_evict_kb":181271,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":-2,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"24","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130732+0000","last_change":"2020-11-18T11:59:16.220885+0000","last_active":"2020-11-18T12:01:41.130732+0000","last_peered":"2020-11-18T12:01:41.130732+0000","last_clean":"2020-11-18T12:01:41.130732+0000","last_became_active":"2020-11-18T11:59:16.220460+0000","last_became_peered":"2020-11-18T11:59:16.220460+0000","last_unstale":"2020-11-18T12:01:41.130732+0000","last_undegraded":"2020-11-18T12:01:41.130732+0000","last_fullsized":"2020-11-18T12:01:41.130732+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"23","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130021+0000","last_change":"2020-11-18T11:59:16.499247+0000","last_active":"2020-11-18T12:01:41.130021+0000","last_peered":"2020-11-18T12:01:41.130021+0000","last_clean":"2020-11-18T12:01:41.130021+0000","last_became_active":"2020-11-18T11:59:16.498717+0000","last_became_peered":"2020-11-18T11:59:16.498717+0000","last_unstale":"2020-11-18T12:01:41.130021+0000","last_undegraded":"2020-11-18T12:01:41.130021+0000","last_fullsized":"2020-11-18T12:01:41.130021+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"34'5488","reported_seq":"7600","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:05.424086+0000","last_change":"2020-11-18T11:59:27.193656+0000","last_active":"2020-11-18T12:02:05.424086+0000","last_peered":"2020-11-18T12:02:05.424086+0000","last_clean":"2020-11-18T12:02:05.424086+0000","last_became_active":"2020-11-18T11:59:27.193348+0000","last_became_peered":"2020-11-18T11:59:27.193348+0000","last_unstale":"2020-11-18T12:02:05.424086+0000","last_undegraded":"2020-11-18T12:02:05.424086+0000","last_fullsized":"2020-11-18T12:02:05.424086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5488,"ondisk_log_size":5488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":133229742,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":1879,"num_read_kb":944,"num_write":4937,"num_write_kb":1137382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":2934,"num_flush":871,"num_flush_kb":2574747,"num_evict":807,"num_evict_kb":2383573,"num_promote":973,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"34'6592","reported_seq":"8537","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.465679+0000","last_change":"2020-11-18T11:59:25.400522+0000","last_active":"2020-11-18T12:02:15.465679+0000","last_peered":"2020-11-18T12:02:15.465679+0000","last_clean":"2020-11-18T12:02:15.465679+0000","last_became_active":"2020-11-18T11:59:25.399994+0000","last_became_peered":"2020-11-18T11:59:25.399994+0000","last_unstale":"2020-11-18T12:02:15.465679+0000","last_undegraded":"2020-11-18T12:02:15.465679+0000","last_fullsized":"2020-11-18T12:02:15.465679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6592,"ondisk_log_size":6592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7292778019,"num_objects":2447,"num_object_clones":0,"num_object_copies":4894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2447,"num_whiteouts":0,"num_read":3524,"num_read_kb":41917,"num_write":8202,"num_write_kb":4621789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"24","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130666+0000","last_change":"2020-11-18T11:59:16.498778+0000","last_active":"2020-11-18T12:01:41.130666+0000","last_peered":"2020-11-18T12:01:41.130666+0000","last_clean":"2020-11-18T12:01:41.130666+0000","last_became_active":"2020-11-18T11:59:16.498407+0000","last_became_peered":"2020-11-18T11:59:16.498407+0000","last_unstale":"2020-11-18T12:01:41.130666+0000","last_undegraded":"2020-11-18T12:01:41.130666+0000","last_fullsized":"2020-11-18T12:01:41.130666+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"34'5230","reported_seq":"7236","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:06.598034+0000","last_change":"2020-11-18T11:59:27.469664+0000","last_active":"2020-11-18T12:02:06.598034+0000","last_peered":"2020-11-18T12:02:06.598034+0000","last_clean":"2020-11-18T12:02:06.598034+0000","last_became_active":"2020-11-18T11:59:27.469083+0000","last_became_peered":"2020-11-18T11:59:27.469083+0000","last_unstale":"2020-11-18T12:02:06.598034+0000","last_undegraded":"2020-11-18T12:02:06.598034+0000","last_fullsized":"2020-11-18T12:02:06.598034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5230,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":139283403,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":1837,"num_read_kb":905,"num_write":4791,"num_write_kb":1124747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":3402,"num_flush":843,"num_flush_kb":2441950,"num_evict":772,"num_evict_kb":2231779,"num_promote":919,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"34'6685","reported_seq":"8719","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.457184+0000","last_change":"2020-11-18T11:59:25.186658+0000","last_active":"2020-11-18T12:02:15.457184+0000","last_peered":"2020-11-18T12:02:15.457184+0000","last_clean":"2020-11-18T12:02:15.457184+0000","last_became_active":"2020-11-18T11:59:25.186414+0000","last_became_peered":"2020-11-18T11:59:25.186414+0000","last_unstale":"2020-11-18T12:02:15.457184+0000","last_undegraded":"2020-11-18T12:02:15.457184+0000","last_fullsized":"2020-11-18T12:02:15.457184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6685,"ondisk_log_size":6685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7453896300,"num_objects":2479,"num_object_clones":0,"num_object_copies":4958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2479,"num_whiteouts":0,"num_read":3574,"num_read_kb":19412,"num_write":8297,"num_write_kb":4811913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"23","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130006+0000","last_change":"2020-11-18T11:59:16.499200+0000","last_active":"2020-11-18T12:01:41.130006+0000","last_peered":"2020-11-18T12:01:41.130006+0000","last_clean":"2020-11-18T12:01:41.130006+0000","last_became_active":"2020-11-18T11:59:16.498419+0000","last_became_peered":"2020-11-18T11:59:16.498419+0000","last_unstale":"2020-11-18T12:01:41.130006+0000","last_undegraded":"2020-11-18T12:01:41.130006+0000","last_fullsized":"2020-11-18T12:01:41.130006+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"34'5183","reported_seq":"7371","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:06.410297+0000","last_change":"2020-11-18T11:59:27.469738+0000","last_active":"2020-11-18T12:02:06.410297+0000","last_peered":"2020-11-18T12:02:06.410297+0000","last_clean":"2020-11-18T12:02:06.410297+0000","last_became_active":"2020-11-18T11:59:27.469241+0000","last_became_peered":"2020-11-18T11:59:27.469241+0000","last_unstale":"2020-11-18T12:02:06.410297+0000","last_undegraded":"2020-11-18T12:02:06.410297+0000","last_fullsized":"2020-11-18T12:02:06.410297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5183,"ondisk_log_size":5183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":150727528,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":1927,"num_read_kb":886,"num_write":4708,"num_write_kb":1109010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":2455,"num_flush":821,"num_flush_kb":2396213,"num_evict":630,"num_evict_kb":1841924,"num_promote":896,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"34'6480","reported_seq":"8496","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.440474+0000","last_change":"2020-11-18T11:59:25.400594+0000","last_active":"2020-11-18T12:02:15.440474+0000","last_peered":"2020-11-18T12:02:15.440474+0000","last_clean":"2020-11-18T12:02:15.440474+0000","last_became_active":"2020-11-18T11:59:25.400132+0000","last_became_peered":"2020-11-18T11:59:25.400132+0000","last_unstale":"2020-11-18T12:02:15.440474+0000","last_undegraded":"2020-11-18T12:02:15.440474+0000","last_fullsized":"2020-11-18T12:02:15.440474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6480,"ondisk_log_size":6480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7283473264,"num_objects":2438,"num_object_clones":0,"num_object_copies":4876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":3498,"num_read_kb":27406,"num_write":8030,"num_write_kb":4749930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.129702+0000","last_change":"2020-11-18T11:59:16.499029+0000","last_active":"2020-11-18T12:01:41.129702+0000","last_peered":"2020-11-18T12:01:41.129702+0000","last_clean":"2020-11-18T12:01:41.129702+0000","last_became_active":"2020-11-18T11:59:16.498581+0000","last_became_peered":"2020-11-18T11:59:16.498581+0000","last_unstale":"2020-11-18T12:01:41.129702+0000","last_undegraded":"2020-11-18T12:01:41.129702+0000","last_fullsized":"2020-11-18T12:01:41.129702+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"34'5553","reported_seq":"7740","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:07.814557+0000","last_change":"2020-11-18T11:59:27.192743+0000","last_active":"2020-11-18T12:02:07.814557+0000","last_peered":"2020-11-18T12:02:07.814557+0000","last_clean":"2020-11-18T12:02:07.814557+0000","last_became_active":"2020-11-18T11:59:27.192298+0000","last_became_peered":"2020-11-18T11:59:27.192298+0000","last_unstale":"2020-11-18T12:02:07.814557+0000","last_undegraded":"2020-11-18T12:02:07.814557+0000","last_fullsized":"2020-11-18T12:02:07.814557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5553,"ondisk_log_size":5553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":139702080,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":1945,"num_read_kb":972,"num_write":5029,"num_write_kb":1151459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":3818,"num_flush":891,"num_flush_kb":2600914,"num_evict":808,"num_evict_kb":2371256,"num_promote":995,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"34'6625","reported_seq":"8418","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.441145+0000","last_change":"2020-11-18T11:59:25.400403+0000","last_active":"2020-11-18T12:02:15.441145+0000","last_peered":"2020-11-18T12:02:15.441145+0000","last_clean":"2020-11-18T12:02:15.441145+0000","last_became_active":"2020-11-18T11:59:25.400017+0000","last_became_peered":"2020-11-18T11:59:25.400017+0000","last_unstale":"2020-11-18T12:02:15.441145+0000","last_undegraded":"2020-11-18T12:02:15.441145+0000","last_fullsized":"2020-11-18T12:02:15.441145+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6625,"ondisk_log_size":6625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7313935544,"num_objects":2442,"num_object_clones":0,"num_object_copies":4884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2442,"num_whiteouts":0,"num_read":3376,"num_read_kb":19124,"num_write":8248,"num_write_kb":4597811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.132577+0000","last_change":"2020-11-18T11:59:16.174660+0000","last_active":"2020-11-18T12:01:41.132577+0000","last_peered":"2020-11-18T12:01:41.132577+0000","last_clean":"2020-11-18T12:01:41.132577+0000","last_became_active":"2020-11-18T11:59:16.174201+0000","last_became_peered":"2020-11-18T11:59:16.174201+0000","last_unstale":"2020-11-18T12:01:41.132577+0000","last_undegraded":"2020-11-18T12:01:41.132577+0000","last_fullsized":"2020-11-18T12:01:41.132577+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.132535+0000","last_change":"2020-11-18T11:59:16.174652+0000","last_active":"2020-11-18T12:01:41.132535+0000","last_peered":"2020-11-18T12:01:41.132535+0000","last_clean":"2020-11-18T12:01:41.132535+0000","last_became_active":"2020-11-18T11:59:16.174176+0000","last_became_peered":"2020-11-18T11:59:16.174176+0000","last_unstale":"2020-11-18T12:01:41.132535+0000","last_undegraded":"2020-11-18T12:01:41.132535+0000","last_fullsized":"2020-11-18T12:01:41.132535+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.129724+0000","last_change":"2020-11-18T11:59:16.220833+0000","last_active":"2020-11-18T12:01:41.129724+0000","last_peered":"2020-11-18T12:01:41.129724+0000","last_clean":"2020-11-18T12:01:41.129724+0000","last_became_active":"2020-11-18T11:59:16.220457+0000","last_became_peered":"2020-11-18T11:59:16.220457+0000","last_unstale":"2020-11-18T12:01:41.129724+0000","last_undegraded":"2020-11-18T12:01:41.129724+0000","last_fullsized":"2020-11-18T12:01:41.129724+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":562942753,"num_objects":215,"num_object_clones":0,"num_object_copies":430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":190,"num_whiteouts":0,"num_read":7588,"num_read_kb":3707,"num_write":19465,"num_write_kb":4522598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":25,"num_bytes_hit_set_archive":12609,"num_flush":3426,"num_flush_kb":10013824,"num_evict":3017,"num_evict_kb":8828532,"num_promote":3783,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":509820928,"data_stored":505904942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21454,"ondisk_log_size":21454,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":4,"stat_sum":{"num_bytes":29344083127,"num_objects":9806,"num_object_clones":0,"num_object_copies":19612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9806,"num_whiteouts":0,"num_read":13972,"num_read_kb":107859,"num_write":32777,"num_write_kb":18781443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":37982961664,"data_stored":37838060392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26382,"ondisk_log_size":26382,"up":8,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640295,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9432696,"kb_used_data":9372000,"kb_used_omap":0,"kb_used_meta":51776,"kb_avail":84939144,"statfs":{"total":96636764160,"available":86977683456,"internally_reserved":0,"allocated":9596928000,"data_stored":9559886176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53018624},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.66200000000000003,"15min":0.64900000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60399999999999998,"15min":0.59399999999999997},"min":{"1min":0.41999999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82499999999999996,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.69699999999999995}]},{"osd":1,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67600000000000005,"15min":0.68200000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.77000000000000002,"15min":0.78300000000000003},"min":{"1min":0.51200000000000001,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.202,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.872}]},{"osd":2,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.80600000000000005,"15min":0.79000000000000004},"min":{"1min":0.52000000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.7010000000000001,"5min":1.7010000000000001,"15min":1.7010000000000001},"last":1.1679999999999999},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.86499999999999999,"15min":0.876},"min":{"1min":0.57099999999999995,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":1.026}]},{"osd":3,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.73499999999999999,"15min":0.73299999999999998},"min":{"1min":0.45100000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.80500000000000005,"15min":0.80900000000000005},"min":{"1min":0.56999999999999995,"5min":0.439,"15min":0.439},"max":{"1min":0.98399999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.98399999999999999}]},{"osd":4,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72299999999999998,"15min":0.74299999999999999},"min":{"1min":0.44800000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.875,"15min":0.876},"min":{"1min":0.64400000000000002,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.504,"5min":1.504,"15min":1.504},"last":1.1950000000000001}]}]},{"osd":4,"up_from":12,"seq":51539607592,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155880,"kb_used_data":119456,"kb_used_omap":0,"kb_used_meta":36416,"kb_avail":94215960,"statfs":{"total":96636764160,"available":96477143040,"internally_reserved":0,"allocated":122322944,"data_stored":121186256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37289984},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.68500000000000005,"15min":0.70599999999999996},"min":{"1min":0.40000000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.81499999999999995,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.64100000000000001,"15min":0.65200000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.878,"5min":1.131,"15min":1.131},"last":0.65300000000000002}]},{"osd":1,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.70099999999999996,"15min":0.70999999999999996},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.97499999999999998,"5min":1.145,"15min":1.145},"last":0.746},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.75700000000000001,"15min":0.77300000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.95599999999999996,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.68700000000000006}]},{"osd":2,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.81999999999999995,"15min":0.83799999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.02,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.84499999999999997,"15min":0.86099999999999999},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.054,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.879}]},{"osd":3,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70099999999999996,"15min":0.70999999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.78500000000000003,"15min":0.78900000000000003},"min":{"1min":0.57999999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.93600000000000005,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.94999999999999996}]},{"osd":5,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72799999999999998,"15min":0.72899999999999998},"min":{"1min":0.57599999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.873,"15min":0.88300000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.0349999999999999,"5min":1.264,"15min":1.264},"last":1.0209999999999999}]}]},{"osd":3,"up_from":12,"seq":51539607591,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4595176,"kb_used_data":4564824,"kb_used_omap":0,"kb_used_meta":28864,"kb_avail":89776664,"statfs":{"total":96636764160,"available":91931303936,"internally_reserved":0,"allocated":4674379776,"data_stored":4655592721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29556736},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.59699999999999998,"15min":0.58699999999999997},"min":{"1min":0.38400000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.421,"5min":1.421,"15min":1.421},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63,"15min":0.63500000000000001},"min":{"1min":0.45000000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91700000000000004,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.89200000000000002}]},{"osd":1,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.68700000000000006,"15min":0.68400000000000005},"min":{"1min":0.496,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.51700000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.7030000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":0.85599999999999998}]},{"osd":2,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.503,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.81399999999999995,"15min":0.83499999999999996},"min":{"1min":0.56599999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":1.0209999999999999}]},{"osd":4,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.46100000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.747},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.78000000000000003,"15min":0.79800000000000004},"min":{"1min":0.49099999999999999,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.046,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.93100000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.82899999999999996,"15min":0.80300000000000005},"min":{"1min":0.56399999999999995,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.137},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.83899999999999997,"15min":0.81499999999999995},"min":{"1min":0.65400000000000003,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.3199999999999998,"5min":2.3199999999999998,"15min":2.3199999999999998},"last":0.997}]}]},{"osd":2,"up_from":12,"seq":51539607591,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9531084,"kb_used_data":9470512,"kb_used_omap":0,"kb_used_meta":51328,"kb_avail":84840756,"statfs":{"total":96636764160,"available":86876934144,"internally_reserved":0,"allocated":9697804288,"data_stored":9661420712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52559872},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.502,"15min":0.5},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.438},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.62,"15min":0.622},"min":{"1min":0.40899999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93799999999999994,"5min":1.464,"15min":1.464},"last":0.48799999999999999}]},{"osd":1,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.68999999999999995,"15min":0.68899999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56799999999999995,"15min":0.57399999999999995},"min":{"1min":0.44500000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.67800000000000005,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.53300000000000003}]},{"osd":3,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.69299999999999995,"15min":0.69799999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0069999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.77000000000000002,"15min":0.78000000000000003},"min":{"1min":0.54300000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.026,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.79200000000000004}]},{"osd":4,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.89400000000000002,"15min":0.89800000000000002},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.101,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.876,"15min":0.88600000000000001},"min":{"1min":0.58899999999999997,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.294,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.75700000000000001}]},{"osd":5,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":2.2210000000000001,"15min":2.4700000000000002},"min":{"1min":0.60699999999999998,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.212,"5min":34.686999999999998,"15min":34.686999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.79800000000000004,"5min":2.266,"15min":2.5110000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.0029999999999999,"5min":34.798000000000002,"15min":34.798000000000002},"last":0.71999999999999997}]}]},{"osd":0,"up_from":12,"seq":51539607591,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155880,"kb_used_data":119456,"kb_used_omap":0,"kb_used_meta":36416,"kb_avail":94215960,"statfs":{"total":96636764160,"available":96477143040,"internally_reserved":0,"allocated":122322944,"data_stored":121186256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37289984},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.64000000000000001,"15min":0.65900000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0529999999999999,"5min":1.754,"15min":1.754},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.57999999999999996,"15min":0.56799999999999995},"min":{"1min":0.41999999999999998,"5min":0.37,"15min":0.37},"max":{"1min":1.639,"5min":1.639,"15min":1.639},"last":0.64100000000000001}]},{"osd":2,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.71499999999999997,"15min":0.72199999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.995,"5min":2.4079999999999999,"15min":2.4079999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.65200000000000002,"15min":0.64700000000000002},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.504}]},{"osd":3,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.76600000000000001,"15min":0.77400000000000002},"min":{"1min":0.51800000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1899999999999999,"5min":1.218,"15min":1.218},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.77500000000000002,"15min":0.79200000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.93600000000000005,"5min":1.181,"15min":1.181},"last":0.60099999999999998}]},{"osd":4,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.89400000000000002,"15min":0.91000000000000003},"min":{"1min":0.56799999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.069,"5min":1.9790000000000001,"15min":1.9790000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.91500000000000004,"15min":0.92700000000000005},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.3200000000000001,"5min":2.141,"15min":2.141},"last":0.92700000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.88700000000000001,"15min":0.88800000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.98699999999999999,"15min":1.002},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.276,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":0.79900000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640296,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14022532,"kb_used_data":13946624,"kb_used_omap":0,"kb_used_meta":73344,"kb_avail":80349308,"statfs":{"total":96636764160,"available":82277691392,"internally_reserved":0,"allocated":14281342976,"data_stored":14225990423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":75104256},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.58699999999999997,"15min":0.58099999999999996},"min":{"1min":0.40799999999999997,"5min":0.373,"15min":0.373},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.56899999999999995,"15min":0.55900000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.68400000000000005}]},{"osd":2,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57799999999999996,"15min":0.57699999999999996},"min":{"1min":0.41599999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.746,"5min":0.70199999999999996,"15min":0.69499999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.115,"5min":1.179,"15min":1.179},"last":0.61499999999999999}]},{"osd":3,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.753,"15min":0.76200000000000001},"min":{"1min":0.46300000000000002,"5min":0.441,"15min":0.441},"max":{"1min":1.1040000000000001,"5min":1.139,"15min":1.139},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.80000000000000004,"15min":0.80200000000000005},"min":{"1min":0.48099999999999998,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.96999999999999997}]},{"osd":4,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.90500000000000003,"15min":0.91700000000000004},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.0700000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":1.117},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.88900000000000001,"15min":0.89800000000000002},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.244,"5min":1.536,"15min":1.536},"last":0.93200000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:02:13 2020","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.93700000000000006,"15min":0.93200000000000005},"min":{"1min":0.70199999999999996,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.387,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.0249999999999999,"5min":0.95299999999999996,"15min":0.94099999999999995},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.48,"5min":1.48,"15min":1.48},"last":1.077}]}]}]}} , 2020-11-18T12:02:21.319 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-11-18T11:59:15.163082+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=11, tm_min=59, tm_sec=15, tm_wday=2, tm_yday=323, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=12, tm_min=2, tm_sec=16, tm_wday=2, tm_yday=323, tm_isdst=0) , 2020-11-18T12:02:21.319 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-11-18T11:59:15.163082+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=11, tm_min=59, tm_sec=15, tm_wday=2, tm_yday=323, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=12, tm_min=2, tm_sec=16, tm_wday=2, tm_yday=323, tm_isdst=0) 2020-11-18T18:42:47.954 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-18T12:03:26.330 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi045.front.sepia.ceph.com , 2020-11-18T12:03:26.330 INFO:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 , 2020-11-18T12:03:26.411 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi072.front.sepia.ceph.com 2020-11-18T18:42:47.954 WARNING:scrape:Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' , 2020-11-18T12:05:24.383 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2020-11-18T18:42:47.955 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status 2020-11-18T18:42:48.068 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-7273-gfbf83bb3 (fbf83bb377e36caf429a163721492f1b1c08deb9) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f14e3a605d4] , 2: (DeleteOp::_begin()+0x519) [0x55ce262f4099] 2020-11-18T18:42:48.070 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-18T12:02:21.315 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2020-11-18T12:02:20.101473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29907025880,"num_objects":10021,"num_object_clones":0,"num_object_copies":20042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9996,"num_whiteouts":0,"num_read":21560,"num_read_kb":111566,"num_write":52242,"num_write_kb":23304041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":25,"num_bytes_hit_set_archive":12609,"num_flush":3426,"num_flush_kb":10013824,"num_evict":3017,"num_evict_kb":8828532,"num_promote":3783,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47836,"ondisk_log_size":47836,"up":32,"acting":32,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":32,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":37893248,"kb_used_data":37592872,"kb_used_omap":0,"kb_used_meta":278144,"kb_avail":528337792,"statfs":{"total":579820584960,"available":541017899008,"internally_reserved":0,"allocated":38495100928,"data_stored":38345262544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":284819456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1626810387,"num_objects":537,"num_object_clones":0,"num_object_copies":1074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":537,"num_whiteouts":0,"num_read":1173,"num_read_kb":102087,"num_write":2624,"num_write_kb":930886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":63,"num_flush_kb":174864,"num_evict":65,"num_evict_kb":181271,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":-2,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"24","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130732+0000","last_change":"2020-11-18T11:59:16.220885+0000","last_active":"2020-11-18T12:01:41.130732+0000","last_peered":"2020-11-18T12:01:41.130732+0000","last_clean":"2020-11-18T12:01:41.130732+0000","last_became_active":"2020-11-18T11:59:16.220460+0000","last_became_peered":"2020-11-18T11:59:16.220460+0000","last_unstale":"2020-11-18T12:01:41.130732+0000","last_undegraded":"2020-11-18T12:01:41.130732+0000","last_fullsized":"2020-11-18T12:01:41.130732+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"23","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130021+0000","last_change":"2020-11-18T11:59:16.499247+0000","last_active":"2020-11-18T12:01:41.130021+0000","last_peered":"2020-11-18T12:01:41.130021+0000","last_clean":"2020-11-18T12:01:41.130021+0000","last_became_active":"2020-11-18T11:59:16.498717+0000","last_became_peered":"2020-11-18T11:59:16.498717+0000","last_unstale":"2020-11-18T12:01:41.130021+0000","last_undegraded":"2020-11-18T12:01:41.130021+0000","last_fullsized":"2020-11-18T12:01:41.130021+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"34'5488","reported_seq":"7600","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:05.424086+0000","last_change":"2020-11-18T11:59:27.193656+0000","last_active":"2020-11-18T12:02:05.424086+0000","last_peered":"2020-11-18T12:02:05.424086+0000","last_clean":"2020-11-18T12:02:05.424086+0000","last_became_active":"2020-11-18T11:59:27.193348+0000","last_became_peered":"2020-11-18T11:59:27.193348+0000","last_unstale":"2020-11-18T12:02:05.424086+0000","last_undegraded":"2020-11-18T12:02:05.424086+0000","last_fullsized":"2020-11-18T12:02:05.424086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5488,"ondisk_log_size":5488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":133229742,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":1879,"num_read_kb":944,"num_write":4937,"num_write_kb":1137382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":2934,"num_flush":871,"num_flush_kb":2574747,"num_evict":807,"num_evict_kb":2383573,"num_promote":973,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"34'6592","reported_seq":"8537","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.465679+0000","last_change":"2020-11-18T11:59:25.400522+0000","last_active":"2020-11-18T12:02:15.465679+0000","last_peered":"2020-11-18T12:02:15.465679+0000","last_clean":"2020-11-18T12:02:15.465679+0000","last_became_active":"2020-11-18T11:59:25.399994+0000","last_became_peered":"2020-11-18T11:59:25.399994+0000","last_unstale":"2020-11-18T12:02:15.465679+0000","last_undegraded":"2020-11-18T12:02:15.465679+0000","last_fullsized":"2020-11-18T12:02:15.465679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6592,"ondisk_log_size":6592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7292778019,"num_objects":2447,"num_object_clones":0,"num_object_copies":4894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2447,"num_whiteouts":0,"num_read":3524,"num_read_kb":41917,"num_write":8202,"num_write_kb":4621789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"24","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130666+0000","last_change":"2020-11-18T11:59:16.498778+0000","last_active":"2020-11-18T12:01:41.130666+0000","last_peered":"2020-11-18T12:01:41.130666+0000","last_clean":"2020-11-18T12:01:41.130666+0000","last_became_active":"2020-11-18T11:59:16.498407+0000","last_became_peered":"2020-11-18T11:59:16.498407+0000","last_unstale":"2020-11-18T12:01:41.130666+0000","last_undegraded":"2020-11-18T12:01:41.130666+0000","last_fullsized":"2020-11-18T12:01:41.130666+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"34'5230","reported_seq":"7236","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:06.598034+0000","last_change":"2020-11-18T11:59:27.469664+0000","last_active":"2020-11-18T12:02:06.598034+0000","last_peered":"2020-11-18T12:02:06.598034+0000","last_clean":"2020-11-18T12:02:06.598034+0000","last_became_active":"2020-11-18T11:59:27.469083+0000","last_became_peered":"2020-11-18T11:59:27.469083+0000","last_unstale":"2020-11-18T12:02:06.598034+0000","last_undegraded":"2020-11-18T12:02:06.598034+0000","last_fullsized":"2020-11-18T12:02:06.598034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5230,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":139283403,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":1837,"num_read_kb":905,"num_write":4791,"num_write_kb":1124747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":3402,"num_flush":843,"num_flush_kb":2441950,"num_evict":772,"num_evict_kb":2231779,"num_promote":919,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"34'6685","reported_seq":"8719","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.457184+0000","last_change":"2020-11-18T11:59:25.186658+0000","last_active":"2020-11-18T12:02:15.457184+0000","last_peered":"2020-11-18T12:02:15.457184+0000","last_clean":"2020-11-18T12:02:15.457184+0000","last_became_active":"2020-11-18T11:59:25.186414+0000","last_became_peered":"2020-11-18T11:59:25.186414+0000","last_unstale":"2020-11-18T12:02:15.457184+0000","last_undegraded":"2020-11-18T12:02:15.457184+0000","last_fullsized":"2020-11-18T12:02:15.457184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6685,"ondisk_log_size":6685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7453896300,"num_objects":2479,"num_object_clones":0,"num_object_copies":4958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2479,"num_whiteouts":0,"num_read":3574,"num_read_kb":19412,"num_write":8297,"num_write_kb":4811913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"23","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130006+0000","last_change":"2020-11-18T11:59:16.499200+0000","last_active":"2020-11-18T12:01:41.130006+0000","last_peered":"2020-11-18T12:01:41.130006+0000","last_clean":"2020-11-18T12:01:41.130006+0000","last_became_active":"2020-11-18T11:59:16.498419+0000","last_became_peered":"2020-11-18T11:59:16.498419+0000","last_unstale":"2020-11-18T12:01:41.130006+0000","last_undegraded":"2020-11-18T12:01:41.130006+0000","last_fullsized":"2020-11-18T12:01:41.130006+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"34'5183","reported_seq":"7371","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:06.410297+0000","last_change":"2020-11-18T11:59:27.469738+0000","last_active":"2020-11-18T12:02:06.410297+0000","last_peered":"2020-11-18T12:02:06.410297+0000","last_clean":"2020-11-18T12:02:06.410297+0000","last_became_active":"2020-11-18T11:59:27.469241+0000","last_became_peered":"2020-11-18T11:59:27.469241+0000","last_unstale":"2020-11-18T12:02:06.410297+0000","last_undegraded":"2020-11-18T12:02:06.410297+0000","last_fullsized":"2020-11-18T12:02:06.410297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5183,"ondisk_log_size":5183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":150727528,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":1927,"num_read_kb":886,"num_write":4708,"num_write_kb":1109010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":2455,"num_flush":821,"num_flush_kb":2396213,"num_evict":630,"num_evict_kb":1841924,"num_promote":896,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"34'6480","reported_seq":"8496","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.440474+0000","last_change":"2020-11-18T11:59:25.400594+0000","last_active":"2020-11-18T12:02:15.440474+0000","last_peered":"2020-11-18T12:02:15.440474+0000","last_clean":"2020-11-18T12:02:15.440474+0000","last_became_active":"2020-11-18T11:59:25.400132+0000","last_became_peered":"2020-11-18T11:59:25.400132+0000","last_unstale":"2020-11-18T12:02:15.440474+0000","last_undegraded":"2020-11-18T12:02:15.440474+0000","last_fullsized":"2020-11-18T12:02:15.440474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6480,"ondisk_log_size":6480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7283473264,"num_objects":2438,"num_object_clones":0,"num_object_copies":4876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":3498,"num_read_kb":27406,"num_write":8030,"num_write_kb":4749930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.129702+0000","last_change":"2020-11-18T11:59:16.499029+0000","last_active":"2020-11-18T12:01:41.129702+0000","last_peered":"2020-11-18T12:01:41.129702+0000","last_clean":"2020-11-18T12:01:41.129702+0000","last_became_active":"2020-11-18T11:59:16.498581+0000","last_became_peered":"2020-11-18T11:59:16.498581+0000","last_unstale":"2020-11-18T12:01:41.129702+0000","last_undegraded":"2020-11-18T12:01:41.129702+0000","last_fullsized":"2020-11-18T12:01:41.129702+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"34'5553","reported_seq":"7740","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:07.814557+0000","last_change":"2020-11-18T11:59:27.192743+0000","last_active":"2020-11-18T12:02:07.814557+0000","last_peered":"2020-11-18T12:02:07.814557+0000","last_clean":"2020-11-18T12:02:07.814557+0000","last_became_active":"2020-11-18T11:59:27.192298+0000","last_became_peered":"2020-11-18T11:59:27.192298+0000","last_unstale":"2020-11-18T12:02:07.814557+0000","last_undegraded":"2020-11-18T12:02:07.814557+0000","last_fullsized":"2020-11-18T12:02:07.814557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5553,"ondisk_log_size":5553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":139702080,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":1945,"num_read_kb":972,"num_write":5029,"num_write_kb":1151459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":3818,"num_flush":891,"num_flush_kb":2600914,"num_evict":808,"num_evict_kb":2371256,"num_promote":995,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"34'6625","reported_seq":"8418","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.441145+0000","last_change":"2020-11-18T11:59:25.400403+0000","last_active":"2020-11-18T12:02:15.441145+0000","last_peered":"2020-11-18T12:02:15.441145+0000","last_clean":"2020-11-18T12:02:15.441145+0000","last_became_active":"2020-11-18T11:59:25.400017+0000","last_became_peered":"2020-11-18T11:59:25.400017+0000","last_unstale":"2020-11-18T12:02:15.441145+0000","last_undegraded":"2020-11-18T12:02:15.441145+0000","last_fullsized":"2020-11-18T12:02:15.441145+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6625,"ondisk_log_size":6625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7313935544,"num_objects":2442,"num_object_clones":0,"num_object_copies":4884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2442,"num_whiteouts":0,"num_read":3376,"num_read_kb":19124,"num_write":8248,"num_write_kb":4597811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.132577+0000","last_change":"2020-11-18T11:59:16.174660+0000","last_active":"2020-11-18T12:01:41.132577+0000","last_peered":"2020-11-18T12:01:41.132577+0000","last_clean":"2020-11-18T12:01:41.132577+0000","last_became_active":"2020-11-18T11:59:16.174201+0000","last_became_peered":"2020-11-18T11:59:16.174201+0000","last_unstale":"2020-11-18T12:01:41.132577+0000","last_undegraded":"2020-11-18T12:01:41.132577+0000","last_fullsized":"2020-11-18T12:01:41.132577+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.132535+0000","last_change":"2020-11-18T11:59:16.174652+0000","last_active":"2020-11-18T12:01:41.132535+0000","last_peered":"2020-11-18T12:01:41.132535+0000","last_clean":"2020-11-18T12:01:41.132535+0000","last_became_active":"2020-11-18T11:59:16.174176+0000","last_became_peered":"2020-11-18T11:59:16.174176+0000","last_unstale":"2020-11-18T12:01:41.132535+0000","last_undegraded":"2020-11-18T12:01:41.132535+0000","last_fullsized":"2020-11-18T12:01:41.132535+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.129724+0000","last_change":"2020-11-18T11:59:16.220833+0000","last_active":"2020-11-18T12:01:41.129724+0000","last_peered":"2020-11-18T12:01:41.129724+0000","last_clean":"2020-11-18T12:01:41.129724+0000","last_became_active":"2020-11-18T11:59:16.220457+0000","last_became_peered":"2020-11-18T11:59:16.220457+0000","last_unstale":"2020-11-18T12:01:41.129724+0000","last_undegraded":"2020-11-18T12:01:41.129724+0000","last_fullsized":"2020-11-18T12:01:41.129724+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":562942753,"num_objects":215,"num_object_clones":0,"num_object_copies":430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":190,"num_whiteouts":0,"num_read":7588,"num_read_kb":3707,"num_write":19465,"num_write_kb":4522598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":25,"num_bytes_hit_set_archive":12609,"num_flush":3426,"num_flush_kb":10013824,"num_evict":3017,"num_evict_kb":8828532,"num_promote":3783,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":509820928,"data_stored":505904942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21454,"ondisk_log_size":21454,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":4,"stat_sum":{"num_bytes":29344083127,"num_objects":9806,"num_object_clones":0,"num_object_copies":19612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9806,"num_whiteouts":0,"num_read":13972,"num_read_kb":107859,"num_write":32777,"num_write_kb":18781443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":37982961664,"data_stored":37838060392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26382,"ondisk_log_size":26382,"up":8,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640295,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9432696,"kb_used_data":9372000,"kb_used_omap":0,"kb_used_meta":51776,"kb_avail":84939144,"statfs":{"total":96636764160,"available":86977683456,"internally_reserved":0,"allocated":9596928000,"data_stored":9559886176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53018624},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.66200000000000003,"15min":0.64900000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60399999999999998,"15min":0.59399999999999997},"min":{"1min":0.41999999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82499999999999996,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.69699999999999995}]},{"osd":1,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67600000000000005,"15min":0.68200000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.77000000000000002,"15min":0.78300000000000003},"min":{"1min":0.51200000000000001,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.202,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.872}]},{"osd":2,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.80600000000000005,"15min":0.79000000000000004},"min":{"1min":0.52000000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.7010000000000001,"5min":1.7010000000000001,"15min":1.7010000000000001},"last":1.1679999999999999},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.86499999999999999,"15min":0.876},"min":{"1min":0.57099999999999995,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":1.026}]},{"osd":3,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.73499999999999999,"15min":0.73299999999999998},"min":{"1min":0.45100000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.80500000000000005,"15min":0.80900000000000005},"min":{"1min":0.56999999999999995,"5min":0.439,"15min":0.439},"max":{"1min":0.98399999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.98399999999999999}]},{"osd":4,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72299999999999998,"15min":0.74299999999999999},"min":{"1min":0.44800000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.875,"15min":0.876},"min":{"1min":0.64400000000000002,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.504,"5min":1.504,"15min":1.504},"last":1.1950000000000001}]}]},{"osd":4,"up_from":12,"seq":51539607592,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155880,"kb_used_data":119456,"kb_used_omap":0,"kb_used_meta":36416,"kb_avail":94215960,"statfs":{"total":96636764160,"available":96477143040,"internally_reserved":0,"allocated":122322944,"data_stored":121186256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37289984},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.68500000000000005,"15min":0.70599999999999996},"min":{"1min":0.40000000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.81499999999999995,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.64100000000000001,"15min":0.65200000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.878,"5min":1.131,"15min":1.131},"last":0.65300000000000002}]},{"osd":1,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.70099999999999996,"15min":0.70999999999999996},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.97499999999999998,"5min":1.145,"15min":1.145},"last":0.746},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.75700000000000001,"15min":0.77300000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.95599999999999996,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.68700000000000006}]},{"osd":2,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.81999999999999995,"15min":0.83799999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.02,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.84499999999999997,"15min":0.86099999999999999},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.054,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.879}]},{"osd":3,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70099999999999996,"15min":0.70999999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.78500000000000003,"15min":0.78900000000000003},"min":{"1min":0.57999999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.93600000000000005,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.94999999999999996}]},{"osd":5,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72799999999999998,"15min":0.72899999999999998},"min":{"1min":0.57599999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.873,"15min":0.88300000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.0349999999999999,"5min":1.264,"15min":1.264},"last":1.0209999999999999}]}]},{"osd":3,"up_from":12,"seq":51539607591,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4595176,"kb_used_data":4564824,"kb_used_omap":0,"kb_used_meta":28864,"kb_avail":89776664,"statfs":{"total":96636764160,"available":91931303936,"internally_reserved":0,"allocated":4674379776,"data_stored":4655592721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29556736},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.59699999999999998,"15min":0.58699999999999997},"min":{"1min":0.38400000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.421,"5min":1.421,"15min":1.421},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63,"15min":0.63500000000000001},"min":{"1min":0.45000000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91700000000000004,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.89200000000000002}]},{"osd":1,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.68700000000000006,"15min":0.68400000000000005},"min":{"1min":0.496,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.51700000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.7030000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":0.85599999999999998}]},{"osd":2,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.503,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.81399999999999995,"15min":0.83499999999999996},"min":{"1min":0.56599999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":1.0209999999999999}]},{"osd":4,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.46100000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.747},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.78000000000000003,"15min":0.79800000000000004},"min":{"1min":0.49099999999999999,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.046,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.93100000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.82899999999999996,"15min":0.80300000000000005},"min":{"1min":0.56399999999999995,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.137},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.83899999999999997,"15min":0.81499999999999995},"min":{"1min":0.65400000000000003,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.3199999999999998,"5min":2.3199999999999998,"15min":2.3199999999999998},"last":0.997}]}]},{"osd":2,"up_from":12,"seq":51539607591,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9531084,"kb_used_data":9470512,"kb_used_omap":0,"kb_used_meta":51328,"kb_avail":84840756,"statfs":{"total":96636764160,"available":86876934144,"internally_reserved":0,"allocated":9697804288,"data_stored":9661420712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52559872},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.502,"15min":0.5},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.438},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.62,"15min":0.622},"min":{"1min":0.40899999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93799999999999994,"5min":1.464,"15min":1.464},"last":0.48799999999999999}]},{"osd":1,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.68999999999999995,"15min":0.68899999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56799999999999995,"15min":0.57399999999999995},"min":{"1min":0.44500000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.67800000000000005,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.53300000000000003}]},{"osd":3,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.69299999999999995,"15min":0.69799999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0069999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.77000000000000002,"15min":0.78000000000000003},"min":{"1min":0.54300000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.026,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.79200000000000004}]},{"osd":4,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.89400000000000002,"15min":0.89800000000000002},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.101,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.876,"15min":0.88600000000000001},"min":{"1min":0.58899999999999997,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.294,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.75700000000000001}]},{"osd":5,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":2.2210000000000001,"15min":2.4700000000000002},"min":{"1min":0.60699999999999998,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.212,"5min":34.686999999999998,"15min":34.686999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.79800000000000004,"5min":2.266,"15min":2.5110000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.0029999999999999,"5min":34.798000000000002,"15min":34.798000000000002},"last":0.71999999999999997}]}]},{"osd":0,"up_from":12,"seq":51539607591,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155880,"kb_used_data":119456,"kb_used_omap":0,"kb_used_meta":36416,"kb_avail":94215960,"statfs":{"total":96636764160,"available":96477143040,"internally_reserved":0,"allocated":122322944,"data_stored":121186256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37289984},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.64000000000000001,"15min":0.65900000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0529999999999999,"5min":1.754,"15min":1.754},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.57999999999999996,"15min":0.56799999999999995},"min":{"1min":0.41999999999999998,"5min":0.37,"15min":0.37},"max":{"1min":1.639,"5min":1.639,"15min":1.639},"last":0.64100000000000001}]},{"osd":2,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.71499999999999997,"15min":0.72199999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.995,"5min":2.4079999999999999,"15min":2.4079999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.65200000000000002,"15min":0.64700000000000002},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.504}]},{"osd":3,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.76600000000000001,"15min":0.77400000000000002},"min":{"1min":0.51800000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1899999999999999,"5min":1.218,"15min":1.218},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.77500000000000002,"15min":0.79200000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.93600000000000005,"5min":1.181,"15min":1.181},"last":0.60099999999999998}]},{"osd":4,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.89400000000000002,"15min":0.91000000000000003},"min":{"1min":0.56799999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.069,"5min":1.9790000000000001,"15min":1.9790000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.91500000000000004,"15min":0.92700000000000005},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.3200000000000001,"5min":2.141,"15min":2.141},"last":0.92700000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.88700000000000001,"15min":0.88800000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.98699999999999999,"15min":1.002},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.276,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":0.79900000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640296,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14022532,"kb_used_data":13946624,"kb_used_omap":0,"kb_used_meta":73344,"kb_avail":80349308,"statfs":{"total":96636764160,"available":82277691392,"internally_reserved":0,"allocated":14281342976,"data_stored":14225990423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":75104256},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.58699999999999997,"15min":0.58099999999999996},"min":{"1min":0.40799999999999997,"5min":0.373,"15min":0.373},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.56899999999999995,"15min":0.55900000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.68400000000000005}]},{"osd":2,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57799999999999996,"15min":0.57699999999999996},"min":{"1min":0.41599999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.746,"5min":0.70199999999999996,"15min":0.69499999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.115,"5min":1.179,"15min":1.179},"last":0.61499999999999999}]},{"osd":3,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.753,"15min":0.76200000000000001},"min":{"1min":0.46300000000000002,"5min":0.441,"15min":0.441},"max":{"1min":1.1040000000000001,"5min":1.139,"15min":1.139},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.80000000000000004,"15min":0.80200000000000005},"min":{"1min":0.48099999999999998,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.96999999999999997}]},{"osd":4,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.90500000000000003,"15min":0.91700000000000004},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.0700000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":1.117},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.88900000000000001,"15min":0.89800000000000002},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.244,"5min":1.536,"15min":1.536},"last":0.93200000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:02:13 2020","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.93700000000000006,"15min":0.93200000000000005},"min":{"1min":0.70199999999999996,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.387,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.0249999999999999,"5min":0.95299999999999996,"15min":0.94099999999999995},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.48,"5min":1.48,"15min":1.48},"last":1.077}]}]}]}} , 2020-11-18T12:02:21.319 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-11-18T11:59:15.163082+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=11, tm_min=59, tm_sec=15, tm_wday=2, tm_yday=323, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=12, tm_min=2, tm_sec=16, tm_wday=2, tm_yday=323, tm_isdst=0) , 2020-11-18T12:02:21.319 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-11-18T11:59:15.163082+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=11, tm_min=59, tm_sec=15, tm_wday=2, tm_yday=323, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=12, tm_min=2, tm_sec=16, tm_wday=2, tm_yday=323, tm_isdst=0) 2020-11-18T18:42:48.071 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-18T12:03:26.330 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi045.front.sepia.ceph.com , 2020-11-18T12:03:26.330 INFO:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 , 2020-11-18T12:03:26.411 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi072.front.sepia.ceph.com 2020-11-18T18:42:48.072 WARNING:scrape:Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' , 2020-11-18T12:05:24.383 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2020-11-18T18:42:48.072 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status 2020-11-18T18:42:48.178 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-7273-gfbf83bb3 (fbf83bb377e36caf429a163721492f1b1c08deb9) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f14e3a605d4] , 2: (DeleteOp::_begin()+0x519) [0x55ce262f4099] 2020-11-18T18:42:48.180 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-18T12:02:21.315 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2020-11-18T12:02:20.101473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29907025880,"num_objects":10021,"num_object_clones":0,"num_object_copies":20042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9996,"num_whiteouts":0,"num_read":21560,"num_read_kb":111566,"num_write":52242,"num_write_kb":23304041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":25,"num_bytes_hit_set_archive":12609,"num_flush":3426,"num_flush_kb":10013824,"num_evict":3017,"num_evict_kb":8828532,"num_promote":3783,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47836,"ondisk_log_size":47836,"up":32,"acting":32,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":32,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":37893248,"kb_used_data":37592872,"kb_used_omap":0,"kb_used_meta":278144,"kb_avail":528337792,"statfs":{"total":579820584960,"available":541017899008,"internally_reserved":0,"allocated":38495100928,"data_stored":38345262544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":284819456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1626810387,"num_objects":537,"num_object_clones":0,"num_object_copies":1074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":537,"num_whiteouts":0,"num_read":1173,"num_read_kb":102087,"num_write":2624,"num_write_kb":930886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":63,"num_flush_kb":174864,"num_evict":65,"num_evict_kb":181271,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":-2,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"24","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130732+0000","last_change":"2020-11-18T11:59:16.220885+0000","last_active":"2020-11-18T12:01:41.130732+0000","last_peered":"2020-11-18T12:01:41.130732+0000","last_clean":"2020-11-18T12:01:41.130732+0000","last_became_active":"2020-11-18T11:59:16.220460+0000","last_became_peered":"2020-11-18T11:59:16.220460+0000","last_unstale":"2020-11-18T12:01:41.130732+0000","last_undegraded":"2020-11-18T12:01:41.130732+0000","last_fullsized":"2020-11-18T12:01:41.130732+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"23","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130021+0000","last_change":"2020-11-18T11:59:16.499247+0000","last_active":"2020-11-18T12:01:41.130021+0000","last_peered":"2020-11-18T12:01:41.130021+0000","last_clean":"2020-11-18T12:01:41.130021+0000","last_became_active":"2020-11-18T11:59:16.498717+0000","last_became_peered":"2020-11-18T11:59:16.498717+0000","last_unstale":"2020-11-18T12:01:41.130021+0000","last_undegraded":"2020-11-18T12:01:41.130021+0000","last_fullsized":"2020-11-18T12:01:41.130021+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"34'5488","reported_seq":"7600","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:05.424086+0000","last_change":"2020-11-18T11:59:27.193656+0000","last_active":"2020-11-18T12:02:05.424086+0000","last_peered":"2020-11-18T12:02:05.424086+0000","last_clean":"2020-11-18T12:02:05.424086+0000","last_became_active":"2020-11-18T11:59:27.193348+0000","last_became_peered":"2020-11-18T11:59:27.193348+0000","last_unstale":"2020-11-18T12:02:05.424086+0000","last_undegraded":"2020-11-18T12:02:05.424086+0000","last_fullsized":"2020-11-18T12:02:05.424086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5488,"ondisk_log_size":5488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":133229742,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":1879,"num_read_kb":944,"num_write":4937,"num_write_kb":1137382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":2934,"num_flush":871,"num_flush_kb":2574747,"num_evict":807,"num_evict_kb":2383573,"num_promote":973,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"34'6592","reported_seq":"8537","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.465679+0000","last_change":"2020-11-18T11:59:25.400522+0000","last_active":"2020-11-18T12:02:15.465679+0000","last_peered":"2020-11-18T12:02:15.465679+0000","last_clean":"2020-11-18T12:02:15.465679+0000","last_became_active":"2020-11-18T11:59:25.399994+0000","last_became_peered":"2020-11-18T11:59:25.399994+0000","last_unstale":"2020-11-18T12:02:15.465679+0000","last_undegraded":"2020-11-18T12:02:15.465679+0000","last_fullsized":"2020-11-18T12:02:15.465679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6592,"ondisk_log_size":6592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7292778019,"num_objects":2447,"num_object_clones":0,"num_object_copies":4894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2447,"num_whiteouts":0,"num_read":3524,"num_read_kb":41917,"num_write":8202,"num_write_kb":4621789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"24","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130666+0000","last_change":"2020-11-18T11:59:16.498778+0000","last_active":"2020-11-18T12:01:41.130666+0000","last_peered":"2020-11-18T12:01:41.130666+0000","last_clean":"2020-11-18T12:01:41.130666+0000","last_became_active":"2020-11-18T11:59:16.498407+0000","last_became_peered":"2020-11-18T11:59:16.498407+0000","last_unstale":"2020-11-18T12:01:41.130666+0000","last_undegraded":"2020-11-18T12:01:41.130666+0000","last_fullsized":"2020-11-18T12:01:41.130666+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"34'5230","reported_seq":"7236","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:06.598034+0000","last_change":"2020-11-18T11:59:27.469664+0000","last_active":"2020-11-18T12:02:06.598034+0000","last_peered":"2020-11-18T12:02:06.598034+0000","last_clean":"2020-11-18T12:02:06.598034+0000","last_became_active":"2020-11-18T11:59:27.469083+0000","last_became_peered":"2020-11-18T11:59:27.469083+0000","last_unstale":"2020-11-18T12:02:06.598034+0000","last_undegraded":"2020-11-18T12:02:06.598034+0000","last_fullsized":"2020-11-18T12:02:06.598034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5230,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":139283403,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":1837,"num_read_kb":905,"num_write":4791,"num_write_kb":1124747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":3402,"num_flush":843,"num_flush_kb":2441950,"num_evict":772,"num_evict_kb":2231779,"num_promote":919,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"34'6685","reported_seq":"8719","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.457184+0000","last_change":"2020-11-18T11:59:25.186658+0000","last_active":"2020-11-18T12:02:15.457184+0000","last_peered":"2020-11-18T12:02:15.457184+0000","last_clean":"2020-11-18T12:02:15.457184+0000","last_became_active":"2020-11-18T11:59:25.186414+0000","last_became_peered":"2020-11-18T11:59:25.186414+0000","last_unstale":"2020-11-18T12:02:15.457184+0000","last_undegraded":"2020-11-18T12:02:15.457184+0000","last_fullsized":"2020-11-18T12:02:15.457184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6685,"ondisk_log_size":6685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7453896300,"num_objects":2479,"num_object_clones":0,"num_object_copies":4958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2479,"num_whiteouts":0,"num_read":3574,"num_read_kb":19412,"num_write":8297,"num_write_kb":4811913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"23","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130006+0000","last_change":"2020-11-18T11:59:16.499200+0000","last_active":"2020-11-18T12:01:41.130006+0000","last_peered":"2020-11-18T12:01:41.130006+0000","last_clean":"2020-11-18T12:01:41.130006+0000","last_became_active":"2020-11-18T11:59:16.498419+0000","last_became_peered":"2020-11-18T11:59:16.498419+0000","last_unstale":"2020-11-18T12:01:41.130006+0000","last_undegraded":"2020-11-18T12:01:41.130006+0000","last_fullsized":"2020-11-18T12:01:41.130006+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"34'5183","reported_seq":"7371","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:06.410297+0000","last_change":"2020-11-18T11:59:27.469738+0000","last_active":"2020-11-18T12:02:06.410297+0000","last_peered":"2020-11-18T12:02:06.410297+0000","last_clean":"2020-11-18T12:02:06.410297+0000","last_became_active":"2020-11-18T11:59:27.469241+0000","last_became_peered":"2020-11-18T11:59:27.469241+0000","last_unstale":"2020-11-18T12:02:06.410297+0000","last_undegraded":"2020-11-18T12:02:06.410297+0000","last_fullsized":"2020-11-18T12:02:06.410297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5183,"ondisk_log_size":5183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":150727528,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":1927,"num_read_kb":886,"num_write":4708,"num_write_kb":1109010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":2455,"num_flush":821,"num_flush_kb":2396213,"num_evict":630,"num_evict_kb":1841924,"num_promote":896,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"34'6480","reported_seq":"8496","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.440474+0000","last_change":"2020-11-18T11:59:25.400594+0000","last_active":"2020-11-18T12:02:15.440474+0000","last_peered":"2020-11-18T12:02:15.440474+0000","last_clean":"2020-11-18T12:02:15.440474+0000","last_became_active":"2020-11-18T11:59:25.400132+0000","last_became_peered":"2020-11-18T11:59:25.400132+0000","last_unstale":"2020-11-18T12:02:15.440474+0000","last_undegraded":"2020-11-18T12:02:15.440474+0000","last_fullsized":"2020-11-18T12:02:15.440474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6480,"ondisk_log_size":6480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7283473264,"num_objects":2438,"num_object_clones":0,"num_object_copies":4876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":3498,"num_read_kb":27406,"num_write":8030,"num_write_kb":4749930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.129702+0000","last_change":"2020-11-18T11:59:16.499029+0000","last_active":"2020-11-18T12:01:41.129702+0000","last_peered":"2020-11-18T12:01:41.129702+0000","last_clean":"2020-11-18T12:01:41.129702+0000","last_became_active":"2020-11-18T11:59:16.498581+0000","last_became_peered":"2020-11-18T11:59:16.498581+0000","last_unstale":"2020-11-18T12:01:41.129702+0000","last_undegraded":"2020-11-18T12:01:41.129702+0000","last_fullsized":"2020-11-18T12:01:41.129702+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"34'5553","reported_seq":"7740","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:07.814557+0000","last_change":"2020-11-18T11:59:27.192743+0000","last_active":"2020-11-18T12:02:07.814557+0000","last_peered":"2020-11-18T12:02:07.814557+0000","last_clean":"2020-11-18T12:02:07.814557+0000","last_became_active":"2020-11-18T11:59:27.192298+0000","last_became_peered":"2020-11-18T11:59:27.192298+0000","last_unstale":"2020-11-18T12:02:07.814557+0000","last_undegraded":"2020-11-18T12:02:07.814557+0000","last_fullsized":"2020-11-18T12:02:07.814557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5553,"ondisk_log_size":5553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":139702080,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":1945,"num_read_kb":972,"num_write":5029,"num_write_kb":1151459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":3818,"num_flush":891,"num_flush_kb":2600914,"num_evict":808,"num_evict_kb":2371256,"num_promote":995,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"34'6625","reported_seq":"8418","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.441145+0000","last_change":"2020-11-18T11:59:25.400403+0000","last_active":"2020-11-18T12:02:15.441145+0000","last_peered":"2020-11-18T12:02:15.441145+0000","last_clean":"2020-11-18T12:02:15.441145+0000","last_became_active":"2020-11-18T11:59:25.400017+0000","last_became_peered":"2020-11-18T11:59:25.400017+0000","last_unstale":"2020-11-18T12:02:15.441145+0000","last_undegraded":"2020-11-18T12:02:15.441145+0000","last_fullsized":"2020-11-18T12:02:15.441145+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6625,"ondisk_log_size":6625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7313935544,"num_objects":2442,"num_object_clones":0,"num_object_copies":4884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2442,"num_whiteouts":0,"num_read":3376,"num_read_kb":19124,"num_write":8248,"num_write_kb":4597811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.132577+0000","last_change":"2020-11-18T11:59:16.174660+0000","last_active":"2020-11-18T12:01:41.132577+0000","last_peered":"2020-11-18T12:01:41.132577+0000","last_clean":"2020-11-18T12:01:41.132577+0000","last_became_active":"2020-11-18T11:59:16.174201+0000","last_became_peered":"2020-11-18T11:59:16.174201+0000","last_unstale":"2020-11-18T12:01:41.132577+0000","last_undegraded":"2020-11-18T12:01:41.132577+0000","last_fullsized":"2020-11-18T12:01:41.132577+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.132535+0000","last_change":"2020-11-18T11:59:16.174652+0000","last_active":"2020-11-18T12:01:41.132535+0000","last_peered":"2020-11-18T12:01:41.132535+0000","last_clean":"2020-11-18T12:01:41.132535+0000","last_became_active":"2020-11-18T11:59:16.174176+0000","last_became_peered":"2020-11-18T11:59:16.174176+0000","last_unstale":"2020-11-18T12:01:41.132535+0000","last_undegraded":"2020-11-18T12:01:41.132535+0000","last_fullsized":"2020-11-18T12:01:41.132535+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.129724+0000","last_change":"2020-11-18T11:59:16.220833+0000","last_active":"2020-11-18T12:01:41.129724+0000","last_peered":"2020-11-18T12:01:41.129724+0000","last_clean":"2020-11-18T12:01:41.129724+0000","last_became_active":"2020-11-18T11:59:16.220457+0000","last_became_peered":"2020-11-18T11:59:16.220457+0000","last_unstale":"2020-11-18T12:01:41.129724+0000","last_undegraded":"2020-11-18T12:01:41.129724+0000","last_fullsized":"2020-11-18T12:01:41.129724+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":562942753,"num_objects":215,"num_object_clones":0,"num_object_copies":430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":190,"num_whiteouts":0,"num_read":7588,"num_read_kb":3707,"num_write":19465,"num_write_kb":4522598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":25,"num_bytes_hit_set_archive":12609,"num_flush":3426,"num_flush_kb":10013824,"num_evict":3017,"num_evict_kb":8828532,"num_promote":3783,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":509820928,"data_stored":505904942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21454,"ondisk_log_size":21454,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":4,"stat_sum":{"num_bytes":29344083127,"num_objects":9806,"num_object_clones":0,"num_object_copies":19612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9806,"num_whiteouts":0,"num_read":13972,"num_read_kb":107859,"num_write":32777,"num_write_kb":18781443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":37982961664,"data_stored":37838060392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26382,"ondisk_log_size":26382,"up":8,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640295,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9432696,"kb_used_data":9372000,"kb_used_omap":0,"kb_used_meta":51776,"kb_avail":84939144,"statfs":{"total":96636764160,"available":86977683456,"internally_reserved":0,"allocated":9596928000,"data_stored":9559886176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53018624},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.66200000000000003,"15min":0.64900000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60399999999999998,"15min":0.59399999999999997},"min":{"1min":0.41999999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82499999999999996,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.69699999999999995}]},{"osd":1,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67600000000000005,"15min":0.68200000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.77000000000000002,"15min":0.78300000000000003},"min":{"1min":0.51200000000000001,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.202,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.872}]},{"osd":2,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.80600000000000005,"15min":0.79000000000000004},"min":{"1min":0.52000000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.7010000000000001,"5min":1.7010000000000001,"15min":1.7010000000000001},"last":1.1679999999999999},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.86499999999999999,"15min":0.876},"min":{"1min":0.57099999999999995,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":1.026}]},{"osd":3,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.73499999999999999,"15min":0.73299999999999998},"min":{"1min":0.45100000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.80500000000000005,"15min":0.80900000000000005},"min":{"1min":0.56999999999999995,"5min":0.439,"15min":0.439},"max":{"1min":0.98399999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.98399999999999999}]},{"osd":4,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72299999999999998,"15min":0.74299999999999999},"min":{"1min":0.44800000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.875,"15min":0.876},"min":{"1min":0.64400000000000002,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.504,"5min":1.504,"15min":1.504},"last":1.1950000000000001}]}]},{"osd":4,"up_from":12,"seq":51539607592,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155880,"kb_used_data":119456,"kb_used_omap":0,"kb_used_meta":36416,"kb_avail":94215960,"statfs":{"total":96636764160,"available":96477143040,"internally_reserved":0,"allocated":122322944,"data_stored":121186256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37289984},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.68500000000000005,"15min":0.70599999999999996},"min":{"1min":0.40000000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.81499999999999995,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.64100000000000001,"15min":0.65200000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.878,"5min":1.131,"15min":1.131},"last":0.65300000000000002}]},{"osd":1,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.70099999999999996,"15min":0.70999999999999996},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.97499999999999998,"5min":1.145,"15min":1.145},"last":0.746},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.75700000000000001,"15min":0.77300000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.95599999999999996,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.68700000000000006}]},{"osd":2,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.81999999999999995,"15min":0.83799999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.02,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.84499999999999997,"15min":0.86099999999999999},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.054,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.879}]},{"osd":3,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70099999999999996,"15min":0.70999999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.78500000000000003,"15min":0.78900000000000003},"min":{"1min":0.57999999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.93600000000000005,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.94999999999999996}]},{"osd":5,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72799999999999998,"15min":0.72899999999999998},"min":{"1min":0.57599999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.873,"15min":0.88300000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.0349999999999999,"5min":1.264,"15min":1.264},"last":1.0209999999999999}]}]},{"osd":3,"up_from":12,"seq":51539607591,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4595176,"kb_used_data":4564824,"kb_used_omap":0,"kb_used_meta":28864,"kb_avail":89776664,"statfs":{"total":96636764160,"available":91931303936,"internally_reserved":0,"allocated":4674379776,"data_stored":4655592721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29556736},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.59699999999999998,"15min":0.58699999999999997},"min":{"1min":0.38400000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.421,"5min":1.421,"15min":1.421},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63,"15min":0.63500000000000001},"min":{"1min":0.45000000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91700000000000004,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.89200000000000002}]},{"osd":1,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.68700000000000006,"15min":0.68400000000000005},"min":{"1min":0.496,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.51700000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.7030000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":0.85599999999999998}]},{"osd":2,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.503,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.81399999999999995,"15min":0.83499999999999996},"min":{"1min":0.56599999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":1.0209999999999999}]},{"osd":4,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.46100000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.747},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.78000000000000003,"15min":0.79800000000000004},"min":{"1min":0.49099999999999999,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.046,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.93100000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.82899999999999996,"15min":0.80300000000000005},"min":{"1min":0.56399999999999995,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.137},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.83899999999999997,"15min":0.81499999999999995},"min":{"1min":0.65400000000000003,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.3199999999999998,"5min":2.3199999999999998,"15min":2.3199999999999998},"last":0.997}]}]},{"osd":2,"up_from":12,"seq":51539607591,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9531084,"kb_used_data":9470512,"kb_used_omap":0,"kb_used_meta":51328,"kb_avail":84840756,"statfs":{"total":96636764160,"available":86876934144,"internally_reserved":0,"allocated":9697804288,"data_stored":9661420712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52559872},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.502,"15min":0.5},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.438},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.62,"15min":0.622},"min":{"1min":0.40899999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93799999999999994,"5min":1.464,"15min":1.464},"last":0.48799999999999999}]},{"osd":1,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.68999999999999995,"15min":0.68899999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56799999999999995,"15min":0.57399999999999995},"min":{"1min":0.44500000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.67800000000000005,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.53300000000000003}]},{"osd":3,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.69299999999999995,"15min":0.69799999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0069999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.77000000000000002,"15min":0.78000000000000003},"min":{"1min":0.54300000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.026,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.79200000000000004}]},{"osd":4,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.89400000000000002,"15min":0.89800000000000002},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.101,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.876,"15min":0.88600000000000001},"min":{"1min":0.58899999999999997,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.294,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.75700000000000001}]},{"osd":5,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":2.2210000000000001,"15min":2.4700000000000002},"min":{"1min":0.60699999999999998,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.212,"5min":34.686999999999998,"15min":34.686999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.79800000000000004,"5min":2.266,"15min":2.5110000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.0029999999999999,"5min":34.798000000000002,"15min":34.798000000000002},"last":0.71999999999999997}]}]},{"osd":0,"up_from":12,"seq":51539607591,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155880,"kb_used_data":119456,"kb_used_omap":0,"kb_used_meta":36416,"kb_avail":94215960,"statfs":{"total":96636764160,"available":96477143040,"internally_reserved":0,"allocated":122322944,"data_stored":121186256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37289984},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.64000000000000001,"15min":0.65900000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0529999999999999,"5min":1.754,"15min":1.754},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.57999999999999996,"15min":0.56799999999999995},"min":{"1min":0.41999999999999998,"5min":0.37,"15min":0.37},"max":{"1min":1.639,"5min":1.639,"15min":1.639},"last":0.64100000000000001}]},{"osd":2,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.71499999999999997,"15min":0.72199999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.995,"5min":2.4079999999999999,"15min":2.4079999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.65200000000000002,"15min":0.64700000000000002},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.504}]},{"osd":3,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.76600000000000001,"15min":0.77400000000000002},"min":{"1min":0.51800000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1899999999999999,"5min":1.218,"15min":1.218},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.77500000000000002,"15min":0.79200000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.93600000000000005,"5min":1.181,"15min":1.181},"last":0.60099999999999998}]},{"osd":4,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.89400000000000002,"15min":0.91000000000000003},"min":{"1min":0.56799999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.069,"5min":1.9790000000000001,"15min":1.9790000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.91500000000000004,"15min":0.92700000000000005},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.3200000000000001,"5min":2.141,"15min":2.141},"last":0.92700000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.88700000000000001,"15min":0.88800000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.98699999999999999,"15min":1.002},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.276,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":0.79900000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640296,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14022532,"kb_used_data":13946624,"kb_used_omap":0,"kb_used_meta":73344,"kb_avail":80349308,"statfs":{"total":96636764160,"available":82277691392,"internally_reserved":0,"allocated":14281342976,"data_stored":14225990423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":75104256},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.58699999999999997,"15min":0.58099999999999996},"min":{"1min":0.40799999999999997,"5min":0.373,"15min":0.373},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.56899999999999995,"15min":0.55900000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.68400000000000005}]},{"osd":2,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57799999999999996,"15min":0.57699999999999996},"min":{"1min":0.41599999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.746,"5min":0.70199999999999996,"15min":0.69499999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.115,"5min":1.179,"15min":1.179},"last":0.61499999999999999}]},{"osd":3,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.753,"15min":0.76200000000000001},"min":{"1min":0.46300000000000002,"5min":0.441,"15min":0.441},"max":{"1min":1.1040000000000001,"5min":1.139,"15min":1.139},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.80000000000000004,"15min":0.80200000000000005},"min":{"1min":0.48099999999999998,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.96999999999999997}]},{"osd":4,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.90500000000000003,"15min":0.91700000000000004},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.0700000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":1.117},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.88900000000000001,"15min":0.89800000000000002},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.244,"5min":1.536,"15min":1.536},"last":0.93200000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:02:13 2020","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.93700000000000006,"15min":0.93200000000000005},"min":{"1min":0.70199999999999996,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.387,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.0249999999999999,"5min":0.95299999999999996,"15min":0.94099999999999995},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.48,"5min":1.48,"15min":1.48},"last":1.077}]}]}]}} , 2020-11-18T12:02:21.319 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-11-18T11:59:15.163082+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=11, tm_min=59, tm_sec=15, tm_wday=2, tm_yday=323, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=12, tm_min=2, tm_sec=16, tm_wday=2, tm_yday=323, tm_isdst=0) , 2020-11-18T12:02:21.319 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-11-18T11:59:15.163082+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=11, tm_min=59, tm_sec=15, tm_wday=2, tm_yday=323, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=12, tm_min=2, tm_sec=16, tm_wday=2, tm_yday=323, tm_isdst=0) 2020-11-18T18:42:48.182 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-18T12:03:26.330 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi045.front.sepia.ceph.com , 2020-11-18T12:03:26.330 INFO:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 , 2020-11-18T12:03:26.411 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi072.front.sepia.ceph.com 2020-11-18T18:42:48.182 WARNING:scrape:Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' , 2020-11-18T12:05:24.383 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2020-11-18T18:42:48.183 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status 2020-11-18T18:42:48.290 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-7273-gfbf83bb3 (fbf83bb377e36caf429a163721492f1b1c08deb9) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f14e3a605d4] , 2: (DeleteOp::_begin()+0x519) [0x55ce262f4099] 2020-11-18T18:42:48.292 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-18T12:02:21.315 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2020-11-18T12:02:20.101473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29907025880,"num_objects":10021,"num_object_clones":0,"num_object_copies":20042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9996,"num_whiteouts":0,"num_read":21560,"num_read_kb":111566,"num_write":52242,"num_write_kb":23304041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":25,"num_bytes_hit_set_archive":12609,"num_flush":3426,"num_flush_kb":10013824,"num_evict":3017,"num_evict_kb":8828532,"num_promote":3783,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47836,"ondisk_log_size":47836,"up":32,"acting":32,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":32,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":37893248,"kb_used_data":37592872,"kb_used_omap":0,"kb_used_meta":278144,"kb_avail":528337792,"statfs":{"total":579820584960,"available":541017899008,"internally_reserved":0,"allocated":38495100928,"data_stored":38345262544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":284819456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1626810387,"num_objects":537,"num_object_clones":0,"num_object_copies":1074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":537,"num_whiteouts":0,"num_read":1173,"num_read_kb":102087,"num_write":2624,"num_write_kb":930886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":63,"num_flush_kb":174864,"num_evict":65,"num_evict_kb":181271,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":-2,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"24","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130732+0000","last_change":"2020-11-18T11:59:16.220885+0000","last_active":"2020-11-18T12:01:41.130732+0000","last_peered":"2020-11-18T12:01:41.130732+0000","last_clean":"2020-11-18T12:01:41.130732+0000","last_became_active":"2020-11-18T11:59:16.220460+0000","last_became_peered":"2020-11-18T11:59:16.220460+0000","last_unstale":"2020-11-18T12:01:41.130732+0000","last_undegraded":"2020-11-18T12:01:41.130732+0000","last_fullsized":"2020-11-18T12:01:41.130732+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"23","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130021+0000","last_change":"2020-11-18T11:59:16.499247+0000","last_active":"2020-11-18T12:01:41.130021+0000","last_peered":"2020-11-18T12:01:41.130021+0000","last_clean":"2020-11-18T12:01:41.130021+0000","last_became_active":"2020-11-18T11:59:16.498717+0000","last_became_peered":"2020-11-18T11:59:16.498717+0000","last_unstale":"2020-11-18T12:01:41.130021+0000","last_undegraded":"2020-11-18T12:01:41.130021+0000","last_fullsized":"2020-11-18T12:01:41.130021+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"34'5488","reported_seq":"7600","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:05.424086+0000","last_change":"2020-11-18T11:59:27.193656+0000","last_active":"2020-11-18T12:02:05.424086+0000","last_peered":"2020-11-18T12:02:05.424086+0000","last_clean":"2020-11-18T12:02:05.424086+0000","last_became_active":"2020-11-18T11:59:27.193348+0000","last_became_peered":"2020-11-18T11:59:27.193348+0000","last_unstale":"2020-11-18T12:02:05.424086+0000","last_undegraded":"2020-11-18T12:02:05.424086+0000","last_fullsized":"2020-11-18T12:02:05.424086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5488,"ondisk_log_size":5488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":133229742,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":1879,"num_read_kb":944,"num_write":4937,"num_write_kb":1137382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":2934,"num_flush":871,"num_flush_kb":2574747,"num_evict":807,"num_evict_kb":2383573,"num_promote":973,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"34'6592","reported_seq":"8537","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.465679+0000","last_change":"2020-11-18T11:59:25.400522+0000","last_active":"2020-11-18T12:02:15.465679+0000","last_peered":"2020-11-18T12:02:15.465679+0000","last_clean":"2020-11-18T12:02:15.465679+0000","last_became_active":"2020-11-18T11:59:25.399994+0000","last_became_peered":"2020-11-18T11:59:25.399994+0000","last_unstale":"2020-11-18T12:02:15.465679+0000","last_undegraded":"2020-11-18T12:02:15.465679+0000","last_fullsized":"2020-11-18T12:02:15.465679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6592,"ondisk_log_size":6592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7292778019,"num_objects":2447,"num_object_clones":0,"num_object_copies":4894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2447,"num_whiteouts":0,"num_read":3524,"num_read_kb":41917,"num_write":8202,"num_write_kb":4621789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"24","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130666+0000","last_change":"2020-11-18T11:59:16.498778+0000","last_active":"2020-11-18T12:01:41.130666+0000","last_peered":"2020-11-18T12:01:41.130666+0000","last_clean":"2020-11-18T12:01:41.130666+0000","last_became_active":"2020-11-18T11:59:16.498407+0000","last_became_peered":"2020-11-18T11:59:16.498407+0000","last_unstale":"2020-11-18T12:01:41.130666+0000","last_undegraded":"2020-11-18T12:01:41.130666+0000","last_fullsized":"2020-11-18T12:01:41.130666+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"34'5230","reported_seq":"7236","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:06.598034+0000","last_change":"2020-11-18T11:59:27.469664+0000","last_active":"2020-11-18T12:02:06.598034+0000","last_peered":"2020-11-18T12:02:06.598034+0000","last_clean":"2020-11-18T12:02:06.598034+0000","last_became_active":"2020-11-18T11:59:27.469083+0000","last_became_peered":"2020-11-18T11:59:27.469083+0000","last_unstale":"2020-11-18T12:02:06.598034+0000","last_undegraded":"2020-11-18T12:02:06.598034+0000","last_fullsized":"2020-11-18T12:02:06.598034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5230,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":139283403,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":1837,"num_read_kb":905,"num_write":4791,"num_write_kb":1124747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":7,"num_bytes_hit_set_archive":3402,"num_flush":843,"num_flush_kb":2441950,"num_evict":772,"num_evict_kb":2231779,"num_promote":919,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"34'6685","reported_seq":"8719","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.457184+0000","last_change":"2020-11-18T11:59:25.186658+0000","last_active":"2020-11-18T12:02:15.457184+0000","last_peered":"2020-11-18T12:02:15.457184+0000","last_clean":"2020-11-18T12:02:15.457184+0000","last_became_active":"2020-11-18T11:59:25.186414+0000","last_became_peered":"2020-11-18T11:59:25.186414+0000","last_unstale":"2020-11-18T12:02:15.457184+0000","last_undegraded":"2020-11-18T12:02:15.457184+0000","last_fullsized":"2020-11-18T12:02:15.457184+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6685,"ondisk_log_size":6685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7453896300,"num_objects":2479,"num_object_clones":0,"num_object_copies":4958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2479,"num_whiteouts":0,"num_read":3574,"num_read_kb":19412,"num_write":8297,"num_write_kb":4811913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"23","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.130006+0000","last_change":"2020-11-18T11:59:16.499200+0000","last_active":"2020-11-18T12:01:41.130006+0000","last_peered":"2020-11-18T12:01:41.130006+0000","last_clean":"2020-11-18T12:01:41.130006+0000","last_became_active":"2020-11-18T11:59:16.498419+0000","last_became_peered":"2020-11-18T11:59:16.498419+0000","last_unstale":"2020-11-18T12:01:41.130006+0000","last_undegraded":"2020-11-18T12:01:41.130006+0000","last_fullsized":"2020-11-18T12:01:41.130006+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"34'5183","reported_seq":"7371","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:06.410297+0000","last_change":"2020-11-18T11:59:27.469738+0000","last_active":"2020-11-18T12:02:06.410297+0000","last_peered":"2020-11-18T12:02:06.410297+0000","last_clean":"2020-11-18T12:02:06.410297+0000","last_became_active":"2020-11-18T11:59:27.469241+0000","last_became_peered":"2020-11-18T11:59:27.469241+0000","last_unstale":"2020-11-18T12:02:06.410297+0000","last_undegraded":"2020-11-18T12:02:06.410297+0000","last_fullsized":"2020-11-18T12:02:06.410297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5183,"ondisk_log_size":5183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":150727528,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":1927,"num_read_kb":886,"num_write":4708,"num_write_kb":1109010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":2455,"num_flush":821,"num_flush_kb":2396213,"num_evict":630,"num_evict_kb":1841924,"num_promote":896,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"34'6480","reported_seq":"8496","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.440474+0000","last_change":"2020-11-18T11:59:25.400594+0000","last_active":"2020-11-18T12:02:15.440474+0000","last_peered":"2020-11-18T12:02:15.440474+0000","last_clean":"2020-11-18T12:02:15.440474+0000","last_became_active":"2020-11-18T11:59:25.400132+0000","last_became_peered":"2020-11-18T11:59:25.400132+0000","last_unstale":"2020-11-18T12:02:15.440474+0000","last_undegraded":"2020-11-18T12:02:15.440474+0000","last_fullsized":"2020-11-18T12:02:15.440474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6480,"ondisk_log_size":6480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7283473264,"num_objects":2438,"num_object_clones":0,"num_object_copies":4876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":3498,"num_read_kb":27406,"num_write":8030,"num_write_kb":4749930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.129702+0000","last_change":"2020-11-18T11:59:16.499029+0000","last_active":"2020-11-18T12:01:41.129702+0000","last_peered":"2020-11-18T12:01:41.129702+0000","last_clean":"2020-11-18T12:01:41.129702+0000","last_became_active":"2020-11-18T11:59:16.498581+0000","last_became_peered":"2020-11-18T11:59:16.498581+0000","last_unstale":"2020-11-18T12:01:41.129702+0000","last_undegraded":"2020-11-18T12:01:41.129702+0000","last_fullsized":"2020-11-18T12:01:41.129702+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"34'5553","reported_seq":"7740","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:07.814557+0000","last_change":"2020-11-18T11:59:27.192743+0000","last_active":"2020-11-18T12:02:07.814557+0000","last_peered":"2020-11-18T12:02:07.814557+0000","last_clean":"2020-11-18T12:02:07.814557+0000","last_became_active":"2020-11-18T11:59:27.192298+0000","last_became_peered":"2020-11-18T11:59:27.192298+0000","last_unstale":"2020-11-18T12:02:07.814557+0000","last_undegraded":"2020-11-18T12:02:07.814557+0000","last_fullsized":"2020-11-18T12:02:07.814557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:26.184115+0000","last_clean_scrub_stamp":"2020-11-18T11:59:26.184115+0000","log_size":5553,"ondisk_log_size":5553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":139702080,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":1945,"num_read_kb":972,"num_write":5029,"num_write_kb":1151459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":3818,"num_flush":891,"num_flush_kb":2600914,"num_evict":808,"num_evict_kb":2371256,"num_promote":995,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"34'6625","reported_seq":"8418","reported_epoch":"34","state":"active+clean","last_fresh":"2020-11-18T12:02:15.441145+0000","last_change":"2020-11-18T11:59:25.400403+0000","last_active":"2020-11-18T12:02:15.441145+0000","last_peered":"2020-11-18T12:02:15.441145+0000","last_clean":"2020-11-18T12:02:15.441145+0000","last_became_active":"2020-11-18T11:59:25.400017+0000","last_became_peered":"2020-11-18T11:59:25.400017+0000","last_unstale":"2020-11-18T12:02:15.441145+0000","last_undegraded":"2020-11-18T12:02:15.441145+0000","last_fullsized":"2020-11-18T12:02:15.441145+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:24.178378+0000","last_clean_scrub_stamp":"2020-11-18T11:59:24.178378+0000","log_size":6625,"ondisk_log_size":6625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7313935544,"num_objects":2442,"num_object_clones":0,"num_object_copies":4884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2442,"num_whiteouts":0,"num_read":3376,"num_read_kb":19124,"num_write":8248,"num_write_kb":4597811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.132577+0000","last_change":"2020-11-18T11:59:16.174660+0000","last_active":"2020-11-18T12:01:41.132577+0000","last_peered":"2020-11-18T12:01:41.132577+0000","last_clean":"2020-11-18T12:01:41.132577+0000","last_became_active":"2020-11-18T11:59:16.174201+0000","last_became_peered":"2020-11-18T11:59:16.174201+0000","last_unstale":"2020-11-18T12:01:41.132577+0000","last_undegraded":"2020-11-18T12:01:41.132577+0000","last_fullsized":"2020-11-18T12:01:41.132577+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.132535+0000","last_change":"2020-11-18T11:59:16.174652+0000","last_active":"2020-11-18T12:01:41.132535+0000","last_peered":"2020-11-18T12:01:41.132535+0000","last_clean":"2020-11-18T12:01:41.132535+0000","last_became_active":"2020-11-18T11:59:16.174176+0000","last_became_peered":"2020-11-18T11:59:16.174176+0000","last_unstale":"2020-11-18T12:01:41.132535+0000","last_undegraded":"2020-11-18T12:01:41.132535+0000","last_fullsized":"2020-11-18T12:01:41.132535+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"25","reported_epoch":"33","state":"active+clean","last_fresh":"2020-11-18T12:01:41.129724+0000","last_change":"2020-11-18T11:59:16.220833+0000","last_active":"2020-11-18T12:01:41.129724+0000","last_peered":"2020-11-18T12:01:41.129724+0000","last_clean":"2020-11-18T12:01:41.129724+0000","last_became_active":"2020-11-18T11:59:16.220457+0000","last_became_peered":"2020-11-18T11:59:16.220457+0000","last_unstale":"2020-11-18T12:01:41.129724+0000","last_undegraded":"2020-11-18T12:01:41.129724+0000","last_fullsized":"2020-11-18T12:01:41.129724+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-18T11:59:15.163082+0000","last_clean_scrub_stamp":"2020-11-18T11:59:15.163082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":562942753,"num_objects":215,"num_object_clones":0,"num_object_copies":430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":190,"num_whiteouts":0,"num_read":7588,"num_read_kb":3707,"num_write":19465,"num_write_kb":4522598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":25,"num_bytes_hit_set_archive":12609,"num_flush":3426,"num_flush_kb":10013824,"num_evict":3017,"num_evict_kb":8828532,"num_promote":3783,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":509820928,"data_stored":505904942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21454,"ondisk_log_size":21454,"up":8,"acting":8,"num_store_stats":5},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":4,"stat_sum":{"num_bytes":29344083127,"num_objects":9806,"num_object_clones":0,"num_object_copies":19612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9806,"num_whiteouts":0,"num_read":13972,"num_read_kb":107859,"num_write":32777,"num_write_kb":18781443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":37982961664,"data_stored":37838060392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26382,"ondisk_log_size":26382,"up":8,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640295,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9432696,"kb_used_data":9372000,"kb_used_omap":0,"kb_used_meta":51776,"kb_avail":84939144,"statfs":{"total":96636764160,"available":86977683456,"internally_reserved":0,"allocated":9596928000,"data_stored":9559886176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53018624},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.66200000000000003,"15min":0.64900000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60399999999999998,"15min":0.59399999999999997},"min":{"1min":0.41999999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82499999999999996,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.69699999999999995}]},{"osd":1,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67600000000000005,"15min":0.68200000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.77000000000000002,"15min":0.78300000000000003},"min":{"1min":0.51200000000000001,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.202,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.872}]},{"osd":2,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.80600000000000005,"15min":0.79000000000000004},"min":{"1min":0.52000000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.7010000000000001,"5min":1.7010000000000001,"15min":1.7010000000000001},"last":1.1679999999999999},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.86499999999999999,"15min":0.876},"min":{"1min":0.57099999999999995,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":1.026}]},{"osd":3,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.73499999999999999,"15min":0.73299999999999998},"min":{"1min":0.45100000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.80500000000000005,"15min":0.80900000000000005},"min":{"1min":0.56999999999999995,"5min":0.439,"15min":0.439},"max":{"1min":0.98399999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.98399999999999999}]},{"osd":4,"last update":"Wed Nov 18 12:02:15 2020","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72299999999999998,"15min":0.74299999999999999},"min":{"1min":0.44800000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.875,"15min":0.876},"min":{"1min":0.64400000000000002,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.504,"5min":1.504,"15min":1.504},"last":1.1950000000000001}]}]},{"osd":4,"up_from":12,"seq":51539607592,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155880,"kb_used_data":119456,"kb_used_omap":0,"kb_used_meta":36416,"kb_avail":94215960,"statfs":{"total":96636764160,"available":96477143040,"internally_reserved":0,"allocated":122322944,"data_stored":121186256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37289984},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.68500000000000005,"15min":0.70599999999999996},"min":{"1min":0.40000000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.81499999999999995,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.64100000000000001,"15min":0.65200000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.878,"5min":1.131,"15min":1.131},"last":0.65300000000000002}]},{"osd":1,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.70099999999999996,"15min":0.70999999999999996},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.97499999999999998,"5min":1.145,"15min":1.145},"last":0.746},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.75700000000000001,"15min":0.77300000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.95599999999999996,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.68700000000000006}]},{"osd":2,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.81999999999999995,"15min":0.83799999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.02,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.84499999999999997,"15min":0.86099999999999999},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.054,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.879}]},{"osd":3,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.70099999999999996,"15min":0.70999999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.78500000000000003,"15min":0.78900000000000003},"min":{"1min":0.57999999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.93600000000000005,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.94999999999999996}]},{"osd":5,"last update":"Wed Nov 18 12:01:14 2020","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72799999999999998,"15min":0.72899999999999998},"min":{"1min":0.57599999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.873,"15min":0.88300000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.0349999999999999,"5min":1.264,"15min":1.264},"last":1.0209999999999999}]}]},{"osd":3,"up_from":12,"seq":51539607591,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4595176,"kb_used_data":4564824,"kb_used_omap":0,"kb_used_meta":28864,"kb_avail":89776664,"statfs":{"total":96636764160,"available":91931303936,"internally_reserved":0,"allocated":4674379776,"data_stored":4655592721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29556736},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.59699999999999998,"15min":0.58699999999999997},"min":{"1min":0.38400000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.421,"5min":1.421,"15min":1.421},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63,"15min":0.63500000000000001},"min":{"1min":0.45000000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.91700000000000004,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.89200000000000002}]},{"osd":1,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.68700000000000006,"15min":0.68400000000000005},"min":{"1min":0.496,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.51700000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.7030000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":0.85599999999999998}]},{"osd":2,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.503,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.81399999999999995,"15min":0.83499999999999996},"min":{"1min":0.56599999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.633,"5min":1.633,"15min":1.633},"last":1.0209999999999999}]},{"osd":4,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.46100000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.747},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.78000000000000003,"15min":0.79800000000000004},"min":{"1min":0.49099999999999999,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.046,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.93100000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:02:16 2020","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.82899999999999996,"15min":0.80300000000000005},"min":{"1min":0.56399999999999995,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.137},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.83899999999999997,"15min":0.81499999999999995},"min":{"1min":0.65400000000000003,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.3199999999999998,"5min":2.3199999999999998,"15min":2.3199999999999998},"last":0.997}]}]},{"osd":2,"up_from":12,"seq":51539607591,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9531084,"kb_used_data":9470512,"kb_used_omap":0,"kb_used_meta":51328,"kb_avail":84840756,"statfs":{"total":96636764160,"available":86876934144,"internally_reserved":0,"allocated":9697804288,"data_stored":9661420712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52559872},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.502,"15min":0.5},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.438},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.62,"15min":0.622},"min":{"1min":0.40899999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93799999999999994,"5min":1.464,"15min":1.464},"last":0.48799999999999999}]},{"osd":1,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.68999999999999995,"15min":0.68899999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56799999999999995,"15min":0.57399999999999995},"min":{"1min":0.44500000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.67800000000000005,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.53300000000000003}]},{"osd":3,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.69299999999999995,"15min":0.69799999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0069999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.77000000000000002,"15min":0.78000000000000003},"min":{"1min":0.54300000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.026,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.79200000000000004}]},{"osd":4,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.89400000000000002,"15min":0.89800000000000002},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.101,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.876,"15min":0.88600000000000001},"min":{"1min":0.58899999999999997,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.294,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.75700000000000001}]},{"osd":5,"last update":"Wed Nov 18 12:01:16 2020","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":2.2210000000000001,"15min":2.4700000000000002},"min":{"1min":0.60699999999999998,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.212,"5min":34.686999999999998,"15min":34.686999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.79800000000000004,"5min":2.266,"15min":2.5110000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.0029999999999999,"5min":34.798000000000002,"15min":34.798000000000002},"last":0.71999999999999997}]}]},{"osd":0,"up_from":12,"seq":51539607591,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":155880,"kb_used_data":119456,"kb_used_omap":0,"kb_used_meta":36416,"kb_avail":94215960,"statfs":{"total":96636764160,"available":96477143040,"internally_reserved":0,"allocated":122322944,"data_stored":121186256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37289984},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.64000000000000001,"15min":0.65900000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0529999999999999,"5min":1.754,"15min":1.754},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.57999999999999996,"15min":0.56799999999999995},"min":{"1min":0.41999999999999998,"5min":0.37,"15min":0.37},"max":{"1min":1.639,"5min":1.639,"15min":1.639},"last":0.64100000000000001}]},{"osd":2,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.71499999999999997,"15min":0.72199999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.995,"5min":2.4079999999999999,"15min":2.4079999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.65200000000000002,"15min":0.64700000000000002},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.504}]},{"osd":3,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.76600000000000001,"15min":0.77400000000000002},"min":{"1min":0.51800000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1899999999999999,"5min":1.218,"15min":1.218},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.77500000000000002,"15min":0.79200000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.93600000000000005,"5min":1.181,"15min":1.181},"last":0.60099999999999998}]},{"osd":4,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.89400000000000002,"15min":0.91000000000000003},"min":{"1min":0.56799999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.069,"5min":1.9790000000000001,"15min":1.9790000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.91500000000000004,"15min":0.92700000000000005},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.3200000000000001,"5min":2.141,"15min":2.141},"last":0.92700000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:01:15 2020","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.88700000000000001,"15min":0.88800000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.98699999999999999,"15min":1.002},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.276,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":0.79900000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640296,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":14022532,"kb_used_data":13946624,"kb_used_omap":0,"kb_used_meta":73344,"kb_avail":80349308,"statfs":{"total":96636764160,"available":82277691392,"internally_reserved":0,"allocated":14281342976,"data_stored":14225990423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":75104256},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.58699999999999997,"15min":0.58099999999999996},"min":{"1min":0.40799999999999997,"5min":0.373,"15min":0.373},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.56899999999999995,"15min":0.55900000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.68400000000000005}]},{"osd":2,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57799999999999996,"15min":0.57699999999999996},"min":{"1min":0.41599999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.746,"5min":0.70199999999999996,"15min":0.69499999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.115,"5min":1.179,"15min":1.179},"last":0.61499999999999999}]},{"osd":3,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.753,"15min":0.76200000000000001},"min":{"1min":0.46300000000000002,"5min":0.441,"15min":0.441},"max":{"1min":1.1040000000000001,"5min":1.139,"15min":1.139},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.80000000000000004,"15min":0.80200000000000005},"min":{"1min":0.48099999999999998,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.96999999999999997}]},{"osd":4,"last update":"Wed Nov 18 12:01:19 2020","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.90500000000000003,"15min":0.91700000000000004},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.0700000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":1.117},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.88900000000000001,"15min":0.89800000000000002},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.244,"5min":1.536,"15min":1.536},"last":0.93200000000000005}]},{"osd":5,"last update":"Wed Nov 18 12:02:13 2020","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.93700000000000006,"15min":0.93200000000000005},"min":{"1min":0.70199999999999996,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.387,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.0249999999999999,"5min":0.95299999999999996,"15min":0.94099999999999995},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.48,"5min":1.48,"15min":1.48},"last":1.077}]}]}]}} , 2020-11-18T12:02:21.319 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-11-18T11:59:15.163082+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=11, tm_min=59, tm_sec=15, tm_wday=2, tm_yday=323, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=12, tm_min=2, tm_sec=16, tm_wday=2, tm_yday=323, tm_isdst=0) , 2020-11-18T12:02:21.319 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-11-18T11:59:15.163082+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=11, tm_min=59, tm_sec=15, tm_wday=2, tm_yday=323, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=18, tm_hour=12, tm_min=2, tm_sec=16, tm_wday=2, tm_yday=323, tm_isdst=0) 2020-11-18T18:42:48.293 WARNING:scrape:Ignoring malparsed backtrace: 2020-11-18T12:03:26.330 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi045.front.sepia.ceph.com , 2020-11-18T12:03:26.330 INFO:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 , 2020-11-18T12:03:26.411 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi072.front.sepia.ceph.com 2020-11-18T18:42:48.294 WARNING:scrape:Ignoring malparsed backtrace: raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' , 2020-11-18T12:05:24.383 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2020-11-18T18:42:48.295 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status 2020-11-18T18:42:48.312 INFO:scrape:Found 11 distinct failure reasons 2020-11-18T18:42:48.313 INFO:scrape:Failure: Command failed (workunit test misc/ver-health.sh) on smithi025 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbf83bb377e36caf429a163721492f1b1c08deb9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh' 2020-11-18T18:42:48.314 INFO:scrape:1 jobs: ['5635980'] 2020-11-18T18:42:48.314 INFO:scrape:suites: ['rados/standalone/{mon_election/classic', 'supported-random-distro$/{ubuntu_latest}', 'workloads/misc}'] 2020-11-18T18:42:48.315 INFO:scrape: 2020-11-18T18:42:48.315 INFO:scrape:Failure: Command failed (workunit test mon/mon-last-epoch-clean.sh) on smithi158 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbf83bb377e36caf429a163721492f1b1c08deb9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mon-last-epoch-clean.sh' 2020-11-18T18:42:48.316 INFO:scrape:2 jobs: ['5635821', '5636005'] 2020-11-18T18:42:48.316 INFO:scrape:suites intersection: ['workloads/mon}'] 2020-11-18T18:42:48.317 INFO:scrape:suites union: ['rados/standalone/{mon_election/classic', 'rados/standalone/{mon_election/connectivity', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/mon}'] 2020-11-18T18:42:48.317 INFO:scrape: 2020-11-18T18:42:48.318 INFO:scrape:Failure: Command failed (workunit test mon/test_mon_osdmap_prune.sh) on smithi158 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbf83bb377e36caf429a163721492f1b1c08deb9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/test_mon_osdmap_prune.sh' 2020-11-18T18:42:48.318 INFO:scrape:1 jobs: ['5635946'] 2020-11-18T18:42:48.319 INFO:scrape:suites: ['clusters/9-mons', 'mon_election/connectivity', 'msgr-failures/mon-delay', 'msgr/async-v1only', 'objectstore/bluestore-low-osd-mem-target', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{rhel_8}', 'thrashers/one', 'workloads/rados_mon_osdmap_prune}'] 2020-11-18T18:42:48.320 INFO:scrape: 2020-11-18T18:42:48.320 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/mon/caps.sh 2020-11-18T18:42:48.321 INFO:scrape:2 jobs: ['5635981', '5635763'] 2020-11-18T18:42:48.321 INFO:scrape:suites intersection: ['clusters/3-mons', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v2only', 'rados', 'rados/monthrash/{ceph', 'workloads/rados_mon_workunits}'] 2020-11-18T18:42:48.322 INFO:scrape:suites union: ['clusters/3-mons', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-stupid', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'thrashers/one', 'thrashers/sync-many', 'workloads/rados_mon_workunits}'] 2020-11-18T18:42:48.322 INFO:scrape: 2020-11-18T18:42:48.323 INFO:scrape:Failure: Command failed (workunit test cephadm/test_adoption.sh) on smithi141 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbf83bb377e36caf429a163721492f1b1c08deb9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' 2020-11-18T18:42:48.324 INFO:scrape:1 jobs: ['5635815'] 2020-11-18T18:42:48.324 INFO:scrape:suites: ['mon_election/classic', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_adoption}'] 2020-11-18T18:42:48.324 INFO:scrape: 2020-11-18T18:42:48.325 INFO:scrape:Failure: Command failed on smithi186 with status 5: 'sudo systemctl stop ceph-cf0a57f2-2993-11eb-b4fd-001a4aab830c@mon.a' 2020-11-18T18:42:48.325 INFO:scrape:3 jobs: ['5635869', '5635775', '5635822'] 2020-11-18T18:42:48.326 INFO:scrape:suites intersection: ['3-start-upgrade', '4-wait', 'fixed-2', 'rados/cephadm/upgrade/{1-start'] 2020-11-18T18:42:48.326 INFO:scrape:suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro$/{centos_latest}', 'distro$/{rhel_latest}', 'distro$/{ubuntu_18.04}', 'fixed-2', 'mon_election/classic}', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start'] 2020-11-18T18:42:48.327 INFO:scrape: 2020-11-18T18:42:48.327 INFO:scrape:Failure: Command failed on smithi027 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:fbf83bb377e36caf429a163721492f1b1c08deb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71e5e2ea-2992-11eb-b4fd-001a4aab830c -- ceph mon dump -f json' 2020-11-18T18:42:48.328 INFO:scrape:1 jobs: ['5635834'] 2020-11-18T18:42:48.328 INFO:scrape:suites: ['rados/cephadm/smoke-roleless/{distro/ubuntu_18.04', 'start}'] 2020-11-18T18:42:48.328 INFO:scrape: 2020-11-18T18:42:48.329 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi173 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbf83bb377e36caf429a163721492f1b1c08deb9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2020-11-18T18:42:48.329 INFO:scrape:2 jobs: ['5635851', '5635712'] 2020-11-18T18:42:48.330 INFO:scrape:suites intersection: ['task/test_cephadm}'] 2020-11-18T18:42:48.330 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{distro/centos_latest', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_cephadm}'] 2020-11-18T18:42:48.331 INFO:scrape: 2020-11-18T18:42:48.331 INFO:scrape:Failure: "2020-11-18T12:40:20.401197+0000 mon.b (mon.0) 11 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,a (MON_DOWN)" in cluster log 2020-11-18T18:42:48.331 INFO:scrape:3 jobs: ['5635942', '5635754', '5635725'] 2020-11-18T18:42:48.332 INFO:scrape:suites intersection: ['msgr/async-v1only', 'rados'] 2020-11-18T18:42:48.332 INFO:scrape:suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/many', 'msgr/async-v1only', 'no_pools', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/multimon/{clusters/21', 'rados/singleton/{all/osd-recovery', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/mon_clock_no_skews}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/set-chunks-read}'] 2020-11-18T18:42:48.333 INFO:scrape: 2020-11-18T18:42:48.333 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2020-11-18T18:42:48.334 INFO:scrape:1 jobs: ['5635876'] 2020-11-18T18:42:48.334 INFO:scrape:suites: ['mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-stupid', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'supported-random-distro$/{rhel_8}}'] 2020-11-18T18:42:48.334 INFO:scrape: 2020-11-18T18:42:48.335 INFO:scrape:Failure: "2020-11-18T12:34:10.767144+0000 osd.7 (osd.7) 21 : cluster [ERR] osd.7 found snap mapper error on pg 3.0s5 oid 3:039c9466:::smithi02526875-43:1e5 snaps missing in mapper, should be: 1d8,1da,1db,1dd,1e5 was r -2...repaired" in cluster log 2020-11-18T18:42:48.335 INFO:scrape:1 jobs: ['5635904'] 2020-11-18T18:42:48.336 INFO:scrape:suites: ['clusters/{fixed-2', 'fast/fast', 'mon_election/classic', 'msgr-failures/osd-delay', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/thrash-erasure-code/{ceph', 'recovery-overrides/{more-async-partial-recovery}', 'supported-random-distro$/{centos_8}', 'thrashers/careful', 'thrashosds-health', 'workloads/ec-rados-plugin=clay-k=4-m=2}'] 2020-11-18T18:42:48.336 INFO:scrape: 2020-11-18T18:42:48.349 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/ failed: 18 dead: 0 running: 0 waiting: 0 queued: 0 passed: 349 Fail ================================================================= [5635712] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/classic task/test_cephadm} ----------------------------------------------------------------- time: 00:10:47 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635712/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635712/ sentry: https://sentry.ceph.com/organizations/ceph/?query=be222d0698d64ce991e387a9a28a5c36 Command failed (workunit test cephadm/test_cephadm.sh) on smithi141 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbf83bb377e36caf429a163721492f1b1c08deb9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [5635725] rados/singleton/{all/osd-recovery mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:10:16 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635725/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635725/ "2020-11-18T10:38:00.455785+0000 mon.a (mon.0) 11 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log [5635754] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} ----------------------------------------------------------------- time: 00:05:55 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635754/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635754/ "2020-11-18T11:02:00.917847+0000 mon.a (mon.0) 23 : cluster [WRN] Health check failed: 1/21 mons down, quorum a,b,c,d,e,f,g,h,i,j,k,l,m,n,p,q,r,s,t,u (MON_DOWN)" in cluster log [5635763] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/one workloads/rados_mon_workunits} ----------------------------------------------------------------- time: 03:24:47 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635763/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635763/ sentry: https://sentry.ceph.com/organizations/ceph/?query=35f36108d2cb419d8331965ec6326e0b Command failed (workunit test mon/caps.sh) on smithi114 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbf83bb377e36caf429a163721492f1b1c08deb9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/caps.sh' [5635775] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{ubuntu_18.04} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:01:59 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635775/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635775/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f1bfcc628f0e4a2f928760284fe6e42c Command failed on smithi146 with status 5: 'sudo systemctl stop ceph-d76e5934-298e-11eb-b4fd-001a4aab830c@mon.a' [5635815] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/classic task/test_adoption} ----------------------------------------------------------------- time: 00:10:44 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635815/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635815/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b42d2102540a4e0b9337b53b8e79524d Command failed (workunit test cephadm/test_adoption.sh) on smithi141 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbf83bb377e36caf429a163721492f1b1c08deb9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' [5635821] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/mon} ----------------------------------------------------------------- time: 00:20:12 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635821/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635821/ sentry: https://sentry.ceph.com/organizations/ceph/?query=32cbcc1e79774ec58e1a3906dbed21d9 Command failed (workunit test mon/mon-last-epoch-clean.sh) on smithi158 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbf83bb377e36caf429a163721492f1b1c08deb9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mon-last-epoch- clean.sh' [5635822] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{rhel_latest} fixed-2 mon_election/classic} ----------------------------------------------------------------- time: 00:06:59 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635822/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635822/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5d79d9a13608439883f178a515502350 Command failed on smithi124 with status 5: 'sudo systemctl stop ceph-9bb0e8be-2991-11eb-b4fd-001a4aab830c@mon.a' [5635834] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04 start} ----------------------------------------------------------------- time: 00:55:11 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635834/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635834/ sentry: https://sentry.ceph.com/organizations/ceph/?query=39ccc58f56f9485ebb29197b4ca78c07 Command failed on smithi027 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:fbf83bb377e36caf429a163721492f1b1c08deb9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 71e5e2ea-2992-11eb-b4fd-001a4aab830c -- ceph mon dump -f json' [5635851] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:09:25 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635851/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635851/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f6f9dd59057a4b978d26b215d208b3c6 Command failed (workunit test cephadm/test_cephadm.sh) on smithi173 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbf83bb377e36caf429a163721492f1b1c08deb9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [5635869] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{centos_latest} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:02:43 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635869/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635869/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0928f06c755f455298a50cad56f58b5b Command failed on smithi186 with status 5: 'sudo systemctl stop ceph- cf0a57f2-2993-11eb-b4fd-001a4aab830c@mon.a' [5635876] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:16:43 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635876/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635876/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' [5635904] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} ----------------------------------------------------------------- time: 00:29:57 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635904/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635904/ "2020-11-18T12:34:10.767144+0000 osd.7 (osd.7) 21 : cluster [ERR] osd.7 found snap mapper error on pg 3.0s5 oid 3:039c9466:::smithi02526875-43:1e5 snaps missing in mapper, should be: 1d8,1da,1db,1dd,1e5 was r -2...repaired" in cluster log [5635942] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/set-chunks-read} ----------------------------------------------------------------- time: 00:13:56 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635942/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635942/ "2020-11-18T12:40:20.401197+0000 mon.b (mon.0) 11 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,a (MON_DOWN)" in cluster log [5635946] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/one workloads/rados_mon_osdmap_prune} ----------------------------------------------------------------- time: 01:05:39 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635946/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635946/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bc3f2adc71c54f9c828273ae891efe07 Command failed (workunit test mon/test_mon_osdmap_prune.sh) on smithi158 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbf83bb377e36caf429a163721492f1b1c08deb9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/mon/test_mon_osdmap_prune.sh' [5635980] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/misc} ----------------------------------------------------------------- time: 00:29:57 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635980/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635980/ sentry: https://sentry.ceph.com/organizations/ceph/?query=da2aa8278f2e4a6c92cc7129bd1c2506 Command failed (workunit test misc/ver-health.sh) on smithi025 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbf83bb377e36caf429a163721492f1b1c08deb9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh' [5635981] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/sync-many workloads/rados_mon_workunits} ----------------------------------------------------------------- time: 03:17:06 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635981/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635981/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b7eeece576e743958101d9954384e3a8 Command failed (workunit test mon/caps.sh) on smithi125 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbf83bb377e36caf429a163721492f1b1c08deb9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/caps.sh' [5636005] rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/mon} ----------------------------------------------------------------- time: 00:20:55 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636005/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636005/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c6a622020b3049c7bee3d8b6f593de20 Command failed (workunit test mon/mon-last-epoch-clean.sh) on smithi122 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbf83bb377e36caf429a163721492f1b1c08deb9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mon-last-epoch- clean.sh' Pass ================================================================= [5635696] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:10 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635696/ [5635697] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} tasks/module_selftest} time: 00:25:32 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635697/ [5635698] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/classic start} time: 00:13:07 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635698/ [5635699] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/small-objects-localized} time: 00:24:50 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635699/ [5635700] rados/singleton/{all/osd-backfill mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} time: 00:10:21 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635700/ [5635701] rados/cephadm/smoke-roleless/{distro/ubuntu_latest start} time: 00:11:01 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635701/ [5635702] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:15:49 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635702/ [5635703] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/root mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:24:49 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635703/ [5635704] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:08:33 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635704/ [5635705] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados tasks/mon_recovery validater/lockdep} time: 00:10:06 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635705/ [5635706] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:10:39 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635706/ [5635707] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/connectivity start} time: 00:13:00 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635707/ [5635708] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} tasks/rados_workunit_loadgen_mix} time: 00:22:07 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635708/ [5635709] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:54 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635709/ [5635710] rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:19:07 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635710/ [5635711] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/small-objects} time: 00:24:42 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635711/ [5635713] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:42:13 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635713/ [5635714] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:11:08 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635714/ [5635715] rados/objectstore/{backends/alloc-hint supported-random-distro$/{rhel_8}} time: 00:15:34 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635715/ [5635716] rados/rest/{mgr-restful supported-random-distro$/{centos_8}} time: 00:10:47 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635716/ [5635717] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:18:36 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635717/ [5635718] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/crush} time: 00:12:17 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635718/ [5635719] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap mon_election/classic thrashosds-health ubuntu_latest} time: 05:07:30 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635719/ [5635720] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:17:37 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635720/ [5635721] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:11:45 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635721/ [5635722] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-radosbench} time: 02:43:11 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635722/ [5635723] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/radosbench} time: 01:39:14 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635723/ [5635724] rados/cephadm/dashboard/{distro/centos_latest task/test_e2e} time: 00:21:28 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635724/ [5635726] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} time: 00:26:06 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635726/ [5635727] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/many workloads/rados_mon_osdmap_prune} time: 01:10:57 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635727/ [5635728] rados/cephadm/orchestrator_cli/{2-node-mgr orchestrator_cli supported-random-distro$/{ubuntu_latest}} time: 00:09:16 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635728/ [5635729] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:15:56 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635729/ [5635730] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/classic start} time: 00:12:04 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635730/ [5635731] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:15:07 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635731/ [5635732] rados/singleton/{all/peer mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:05:54 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635732/ [5635733] rados/cephadm/smoke-roleless/{distro/centos_8.0 start} time: 00:12:40 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635733/ [5635734] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:07:56 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635734/ [5635735] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/filestore-xfs supported-random-distro$/{centos_8} tasks/progress} time: 00:11:39 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635735/ [5635736] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-localized} time: 00:24:01 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635736/ [5635737] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:09:40 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635737/ [5635738] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{ubuntu_18.04_podman} fixed-2 mon_election/classic} time: 00:19:05 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635738/ [5635739] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:06:05 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635739/ [5635740] rados/cephadm/with-work/{distro/ubuntu_18.04_podman fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} time: 00:37:45 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635740/ [5635741] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_mostlyread} time: 00:37:10 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635741/ [5635742] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/erasure-code} time: 00:44:38 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635742/ [5635743] rados/singleton/{all/pg-autoscaler-progress-off mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}} time: 00:11:32 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635743/ [5635744] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/few rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:25:46 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635744/ [5635745] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/pggrow thrashosds-health workloads/rbd_cls} time: 00:20:25 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635745/ [5635746] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:26:26 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635746/ [5635747] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/valgrind} time: 00:50:54 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635747/ [5635748] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:28:31 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635748/ [5635749] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:29:21 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635749/ [5635750] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/connectivity start} time: 00:10:24 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635750/ [5635751] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:09:49 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635751/ [5635752] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_cephadm_repos} time: 00:07:04 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635752/ [5635753] rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8}} time: 00:11:59 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635753/ [5635755] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-small-objects-balanced} time: 00:25:13 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635755/ [5635756] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/classic start} time: 00:15:36 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635756/ [5635757] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:10:48 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635757/ [5635758] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:04:17 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635758/ [5635759] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{ubuntu_latest}} time: 00:09:15 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635759/ [5635760] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/write_fadvise_dontneed} time: 00:10:30 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635760/ [5635761] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:09:11 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635761/ [5635762] rados/cephadm/smoke-roleless/{distro/centos_latest start} time: 00:12:21 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635762/ [5635764] rados/singleton/{all/pg-removal-interruption mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} time: 00:11:26 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635764/ [5635765] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/root mon_election/classic msgr/async-v2only start tasks/rados_python} time: 00:24:26 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635765/ [5635766] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:15:08 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635766/ [5635767] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/careful thrashosds-health workloads/snaps-few-objects} time: 00:38:21 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635767/ [5635768] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/prometheus} time: 00:09:07 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635768/ [5635769] rados/standalone/{mon_election/classic supported-random-distro$/{rhel_8} workloads/mgr} time: 00:17:44 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635769/ [5635770] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/admin_socket_objecter_requests} time: 00:31:27 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635770/ [5635771] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:04:37 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635771/ [5635772] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/connectivity start} time: 00:14:47 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635772/ [5635773] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} tasks/readwrite} time: 00:09:22 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635773/ [5635774] rados/singleton/{all/radostool mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} time: 00:14:29 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635774/ [5635776] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} tasks/dashboard} time: 01:05:22 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635776/ [5635777] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/default thrashosds-health workloads/test_rbd_api} time: 00:19:07 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635777/ [5635778] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:08:33 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635778/ [5635779] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:11:27 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635779/ [5635780] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/cache-agent-big} time: 00:22:49 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635780/ [5635781] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/classic start} time: 00:12:32 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635781/ [5635782] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:08:25 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635782/ [5635783] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados tasks/rados_cls_all validater/lockdep} time: 00:19:38 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635783/ [5635784] rados/singleton/{all/random-eio mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} time: 00:29:04 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635784/ [5635785] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:28:27 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635785/ [5635786] rados/cephadm/smoke-roleless/{distro/rhel_8.0 start} time: 00:16:13 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635786/ [5635787] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/packaged mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:32:31 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635787/ [5635788] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:04:30 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635788/ [5635789] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-fast-read} time: 00:17:15 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635789/ [5635790] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:11:10 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635790/ [5635791] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/cache-agent-small} time: 00:13:16 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635791/ [5635792] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/classic task/test_orch_cli} time: 00:15:56 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635792/ [5635793] rados/singleton/{all/rebuild-mondb mon_election/classic msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} time: 00:10:08 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635793/ [5635794] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/misc} time: 00:35:05 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635794/ [5635795] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:27:43 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635795/ [5635796] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/connectivity start} time: 00:13:07 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635796/ [5635797] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} tasks/dashboard} time: 01:04:59 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635797/ [5635798] rados/objectstore/{backends/filejournal supported-random-distro$/{ubuntu_latest}} time: 00:05:04 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635798/ [5635799] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:33 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635799/ [5635800] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/sync-many workloads/snaps-few-objects} time: 00:32:39 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635800/ [5635801] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/mapgap thrashosds-health workloads/cache-snaps} time: 00:31:59 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635801/ [5635802] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:10:08 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635802/ [5635803] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} tasks/repair_test} time: 00:14:07 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635803/ [5635804] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:10:06 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635804/ [5635805] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8} tasks/workunits} time: 00:10:04 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635805/ [5635806] rados/singleton/{all/recovery-preemption mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} time: 00:22:59 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635806/ [5635807] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:24:16 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635807/ [5635808] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/classic start} time: 00:12:43 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635808/ [5635809] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:35:49 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635809/ [5635810] rados/cephadm/smoke-roleless/{distro/rhel_latest start} time: 00:16:16 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635810/ [5635811] rados/cephadm/with-work/{distro/rhel_8.0 fixed-2 mode/root mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:28:27 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635811/ [5635812] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:07:33 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635812/ [5635813] rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:06:01 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635813/ [5635814] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps} time: 00:27:35 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635814/ [5635816] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-hybrid supported-random-distro$/{centos_8} tasks/dashboard} time: 01:05:06 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635816/ [5635817] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:25:52 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635817/ [5635818] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados tasks/mon_recovery validater/valgrind} time: 00:26:28 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635818/ [5635819] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/connectivity start} time: 00:12:12 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635819/ [5635820] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:04 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635820/ [5635823] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:11:09 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635823/ [5635824] rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:11:06 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635824/ [5635825] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:08:44 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635825/ [5635826] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} tasks/mon_recovery} time: 00:10:01 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635826/ [5635827] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:21:54 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635827/ [5635828] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-snaps-balanced} time: 00:15:34 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635828/ [5635829] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/morepggrow thrashosds-health workloads/radosbench} time: 01:42:03 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635829/ [5635830] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/classic start} time: 00:10:14 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635830/ [5635831] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:03 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635831/ [5635832] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} tasks/scrub_test} time: 00:14:16 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635832/ [5635833] rados/singleton/{all/test_envlibrados_for_rocksdb mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}} time: 00:16:21 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635833/ [5635835] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:11:33 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635835/ [5635836] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/pool-create-delete} time: 00:13:28 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635836/ [5635837] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8} tasks/crash} time: 00:08:34 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635837/ [5635838] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache-snaps} time: 00:27:02 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635838/ [5635839] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-low-osd-mem-target supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:11:07 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635839/ [5635840] rados/objectstore/{backends/filestore-idempotent-aio-journal supported-random-distro$/{ubuntu_latest}} time: 02:25:28 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635840/ [5635841] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} time: 00:17:54 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635841/ [5635842] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} time: 00:38:18 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635842/ [5635843] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/none thrashosds-health workloads/rbd_cls} time: 00:15:42 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635843/ [5635844] rados/singleton/{all/thrash-backfill-full mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}} time: 01:24:38 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635844/ [5635845] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:18:52 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635845/ [5635846] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/connectivity start} time: 00:14:52 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635846/ [5635847] rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/osd} time: 03:06:50 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635847/ [5635848] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:09:09 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635848/ [5635849] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-small-objects-overwrites} time: 00:26:51 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635849/ [5635850] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/cache} time: 00:10:15 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635850/ [5635852] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/classic start} time: 00:14:21 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635852/ [5635853] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:09:16 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635853/ [5635854] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados tasks/rados_api_tests validater/lockdep} time: 00:27:43 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635854/ [5635855] rados/singleton/{all/thrash-eio mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:31:12 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635855/ [5635856] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:15:00 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635856/ [5635857] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:29:21 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635857/ [5635858] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04_podman start} time: 00:15:47 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635858/ [5635859] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:06:13 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635859/ [5635860] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/pggrow thrashosds-health workloads/snaps-few-objects} time: 00:35:13 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635860/ [5635861] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/pool-snaps-few-objects} time: 00:35:30 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635861/ [5635862] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} tasks/mon_clock_no_skews} time: 00:11:00 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635862/ [5635863] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-small-objects} time: 00:27:28 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635863/ [5635864] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} tasks/rados_api_tests} time: 00:31:19 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635864/ [5635865] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:26:57 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635865/ [5635866] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_python} time: 00:24:11 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635866/ [5635867] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/filestore-xfs rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:13:54 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635867/ [5635868] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:14:49 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635868/ [5635870] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:09:00 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635870/ [5635871] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_5925} time: 00:08:06 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635871/ [5635872] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/connectivity start} time: 00:14:06 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635872/ [5635873] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/rados_api_tests} time: 00:24:45 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635873/ [5635874] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} tasks/failover} time: 00:09:39 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635874/ [5635875] rados/standalone/{mon_election/classic supported-random-distro$/{rhel_8} workloads/scrub} time: 01:46:01 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635875/ [5635877] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/careful thrashosds-health workloads/test_rbd_api} time: 00:20:26 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635877/ [5635878] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:26:23 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635878/ [5635879] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:04:09 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635879/ [5635880] rados/objectstore/{backends/filestore-idempotent supported-random-distro$/{centos_8}} time: 02:16:09 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635880/ [5635881] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/classic start} time: 00:13:24 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635881/ [5635882] rados/cephadm/smoke-roleless/{distro/ubuntu_latest start} time: 00:11:30 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635882/ [5635883] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/radosbench-high-concurrency} time: 00:37:11 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635883/ [5635884] rados/singleton/{all/watch-notify-same-primary mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8}} time: 00:05:47 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635884/ [5635885] rados/cephadm/with-work/{distro/ubuntu_18.04_podman fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} time: 00:35:49 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635885/ [5635886] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:10:14 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635886/ [5635887] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:31:26 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635887/ [5635888] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados tasks/rados_cls_all validater/valgrind} time: 02:11:07 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635888/ [5635889] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/classic task/test_cephadm_repos} time: 00:07:02 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635889/ [5635890] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:32:54 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635890/ [5635891] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:07:06 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635891/ [5635892] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} tasks/rados_cls_all} time: 00:23:00 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635892/ [5635893] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/connectivity start} time: 00:13:45 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635893/ [5635894] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/radosbench} time: 01:50:31 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635894/ [5635895] rados/singleton/{all/admin-socket mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} time: 00:11:11 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635895/ [5635896] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/default thrashosds-health workloads/cache-snaps} time: 00:30:59 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635896/ [5635897] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:41:44 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635897/ [5635898] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:06:39 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635898/ [5635899] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:12:22 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635899/ [5635900] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/crush} time: 00:11:56 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635900/ [5635901] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:11 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635901/ [5635902] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap mon_election/connectivity thrashosds-health ubuntu_latest} time: 04:29:34 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635902/ [5635903] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} tasks/mon_clock_with_skews} time: 00:10:48 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635903/ [5635905] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/few rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:21:56 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635905/ [5635906] rados/cephadm/dashboard/{distro/centos_latest task/test_e2e} time: 00:21:52 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635906/ [5635907] rados/singleton/{all/deduptool mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:15:11 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635907/ [5635908] rados/cephadm/orchestrator_cli/{2-node-mgr orchestrator_cli supported-random-distro$/{centos_8}} time: 00:10:21 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635908/ [5635909] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/redirect} time: 00:09:29 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635909/ [5635910] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} tasks/insights} time: 00:09:52 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635910/ [5635911] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/many workloads/rados_api_tests} time: 00:31:25 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635911/ [5635912] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/classic start} time: 00:12:33 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635912/ [5635913] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:10:59 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635913/ [5635914] rados/cephadm/smoke-roleless/{distro/centos_8.0 start} time: 00:10:55 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635914/ [5635915] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:20:21 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635915/ [5635916] rados/singleton/{all/divergent_priors mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:06:30 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635916/ [5635917] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{centos_8.0} fixed-2 mon_election/classic} time: 00:15:16 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635917/ [5635918] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/redirect_promote_tests} time: 00:11:06 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635918/ [5635919] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:12:28 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635919/ [5635920] rados/objectstore/{backends/fusestore supported-random-distro$/{rhel_8}} time: 00:14:03 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635920/ [5635921] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:05:49 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635921/ [5635922] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} time: 00:35:06 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635922/ [5635923] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/rados_python} time: 00:15:22 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635923/ [5635924] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/mapgap thrashosds-health workloads/radosbench} time: 01:33:49 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635924/ [5635925] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/erasure-code} time: 00:37:12 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635925/ [5635926] rados/singleton/{all/divergent_priors2 mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:06:36 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635926/ [5635927] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:14:39 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635927/ [5635928] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados tasks/mon_recovery validater/lockdep} time: 00:09:39 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635928/ [5635929] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_orch_cli} time: 00:15:42 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635929/ [5635930] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:20:16 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635930/ [5635931] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/redirect_set_object} time: 00:15:58 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635931/ [5635932] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:09:59 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635932/ [5635933] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/connectivity start} time: 00:10:47 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635933/ [5635934] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:14:28 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635934/ [5635935] rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}} time: 00:12:09 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635935/ [5635936] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} tasks/mon_recovery} time: 00:08:24 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635936/ [5635937] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:24:59 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635937/ [5635938] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python} time: 00:23:37 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635938/ [5635939] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/sample_fio} time: 00:10:58 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635939/ [5635940] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/classic start} time: 00:15:23 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635940/ [5635941] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:06:44 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635941/ [5635943] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-hybrid supported-random-distro$/{centos_8} tasks/module_selftest} time: 00:26:57 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635943/ [5635944] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:08:44 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635944/ [5635945] rados/cephadm/smoke-roleless/{distro/centos_latest start} time: 00:09:09 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635945/ [5635947] rados/singleton/{all/ec-lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} time: 00:50:45 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635947/ [5635948] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/morepggrow thrashosds-health workloads/rbd_cls} time: 00:22:17 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635948/ [5635949] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/connectivity start} time: 00:15:22 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635949/ [5635950] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/mgr} time: 00:11:51 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635950/ [5635951] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/small-objects-balanced} time: 00:22:32 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635951/ [5635952] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:28:07 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635952/ [5635953] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:11:08 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635953/ [5635954] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} tasks/rados_stress_watch} time: 00:17:48 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635954/ [5635955] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_adoption} time: 00:09:54 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635955/ [5635956] rados/singleton/{all/erasure-code-nonregression mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} time: 00:08:36 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635956/ [5635957] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{ubuntu_latest} fixed-2 mon_election/connectivity} time: 00:16:57 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635957/ [5635958] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} tasks/dashboard} time: 01:07:01 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635958/ [5635959] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{ubuntu_latest}} time: 00:11:22 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635959/ [5635960] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} time: 00:19:28 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635960/ [5635961] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:08:29 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635961/ [5635962] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/none thrashosds-health workloads/snaps-few-objects} time: 00:30:46 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635962/ [5635963] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:22:38 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635963/ [5635964] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados tasks/rados_api_tests validater/valgrind} time: 00:53:55 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635964/ [5635965] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/small-objects-localized} time: 00:17:54 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635965/ [5635966] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:06:28 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635966/ [5635967] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:24:47 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635967/ [5635968] rados/cephadm/with-work/{distro/rhel_8.0 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:39:49 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635968/ [5635969] rados/singleton/{all/lost-unfound-delete mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} time: 00:43:19 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635969/ [5635970] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:10 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635970/ [5635971] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/classic start} time: 00:13:06 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635971/ [5635972] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews} time: 00:05:48 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635972/ [5635973] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:25:15 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635973/ [5635974] rados/cephadm/smoke-roleless/{distro/rhel_8.0 start} time: 00:13:56 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635974/ [5635975] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:10:48 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635975/ [5635976] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/small-objects} time: 00:19:12 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635976/ [5635977] rados/singleton/{all/lost-unfound mon_election/classic msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{centos_8}} time: 00:26:30 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635977/ [5635978] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} tasks/progress} time: 00:16:33 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635978/ [5635979] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/connectivity start} time: 00:13:23 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635979/ [5635982] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} tasks/dashboard} time: 01:07:23 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635982/ [5635983] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} time: 00:19:27 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635983/ [5635984] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/cosbench_64K_read_write} time: 00:15:22 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635984/ [5635985] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:28:26 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635985/ [5635986] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} tasks/rados_striper} time: 00:12:25 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635986/ [5635987] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:15:35 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635987/ [5635988] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/snaps-few-objects-balanced} time: 00:24:24 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635988/ [5635989] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:05:05 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635989/ [5635990] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/classic start} time: 00:13:23 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635990/ [5635991] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:36:18 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635991/ [5635992] rados/cephadm/smoke-roleless/{distro/rhel_latest start} time: 00:13:39 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635992/ [5635993] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:07:01 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635993/ [5635994] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_cephadm} time: 00:15:12 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635994/ [5635995] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:09:07 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635995/ [5635996] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/snaps-few-objects-localized} time: 00:29:32 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635996/ [5635997] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:10:23 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635997/ [5635998] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/lockdep} time: 00:24:36 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635998/ [5635999] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/careful thrashosds-health workloads/cache-snaps} time: 00:30:14 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5635999/ [5636000] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:05:34 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636000/ [5636001] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{rhel_8}} time: 00:30:04 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636001/ [5636002] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:28:55 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636002/ [5636003] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:40 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636003/ [5636004] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/connectivity start} time: 00:12:31 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636004/ [5636006] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/fastclose rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:24:31 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636006/ [5636007] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/cosbench_64K_write} time: 00:18:47 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636007/ [5636008] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:10:13 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636008/ [5636009] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:16:45 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636009/ [5636010] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{ubuntu_18.04_podman} fixed-2 mon_election/classic} time: 00:19:01 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636010/ [5636011] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/filestore-xfs rados supported-random-distro$/{centos_8} tasks/mon_clock_with_skews} time: 00:05:31 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636011/ [5636012] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-radosbench} time: 01:15:37 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636012/ [5636013] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects} time: 00:22:21 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636013/ [5636014] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} time: 00:35:11 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636014/ [5636015] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-stupid supported-random-distro$/{centos_8} tasks/prometheus} time: 00:09:20 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636015/ [5636016] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/classic start} time: 00:10:38 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636016/ [5636017] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} tasks/rados_workunit_loadgen_big} time: 00:32:16 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636017/ [5636018] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/sync workloads/snaps-few-objects} time: 00:32:05 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636018/ [5636019] rados/singleton/{all/mon-auth-caps mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} time: 00:15:56 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636019/ [5636020] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:16:49 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636020/ [5636021] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04 start} time: 00:11:29 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636021/ [5636022] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/write_fadvise_dontneed} time: 00:12:31 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636022/ [5636023] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} tasks/dashboard} time: 01:06:37 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636023/ [5636024] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/default thrashosds-health workloads/radosbench} time: 01:25:16 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636024/ [5636025] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_cephadm_repos} time: 00:07:18 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636025/ [5636026] rados/singleton/{all/mon-config-key-caps mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}} time: 00:22:16 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636026/ [5636027] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:19:25 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636027/ [5636028] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:09:52 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636028/ [5636029] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/osd} time: 03:06:54 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636029/ [5636030] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/connectivity start} time: 00:16:03 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636030/ [5636031] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/admin_socket_objecter_requests} time: 00:21:39 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636031/ [5636032] rados/cephadm/with-work/{distro/ubuntu_18.04_podman fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_python} time: 00:25:47 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636032/ [5636033] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:15:34 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636033/ [5636034] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:30:54 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636034/ [5636035] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados tasks/rados_cls_all validater/valgrind} time: 02:07:42 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636035/ [5636036] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:43 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636036/ [5636037] rados/singleton/{all/mon-config-keys mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} time: 00:21:49 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636037/ [5636038] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/mapgap thrashosds-health workloads/rbd_cls} time: 00:25:00 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636038/ [5636039] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:16:06 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636039/ [5636040] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/classic start} time: 00:14:56 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636040/ [5636041] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-stupid supported-random-distro$/{centos_8} tasks/dashboard} time: 01:06:17 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636041/ [5636042] rados/objectstore/{backends/objectstore supported-random-distro$/{ubuntu_latest}} time: 05:11:50 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636042/ [5636043] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache-agent-big} time: 00:21:37 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636043/ [5636044] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} tasks/mon_clock_with_skews} time: 00:11:04 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636044/ [5636045] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-balanced} time: 00:20:18 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636045/ [5636046] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04_podman start} time: 00:11:19 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636046/ [5636047] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_mix} time: 00:21:06 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636047/ [5636048] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} tasks/workunits} time: 00:09:48 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636048/ [5636049] rados/singleton/{all/mon-config mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:10:18 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636049/ [5636050] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{ubuntu_18.04_podman} fixed-2 mon_election/connectivity} time: 00:18:42 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636050/ [5636051] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:10:24 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636051/ [5636052] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:10:39 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636052/ [5636053] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/connectivity start} time: 00:13:06 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636053/ [5636054] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/snaps-few-objects} time: 00:23:17 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636054/ [5636055] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/cache-agent-small} time: 00:17:35 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636055/ [5636056] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/scrub} time: 01:45:21 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636056/ [5636057] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_orch_cli} time: 00:15:54 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636057/ [5636058] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-small-objects-overwrites} time: 00:19:02 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636058/ [5636059] rados/singleton/{all/osd-backfill mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:21:58 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636059/ [5636060] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:36:56 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636060/ [5636061] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:25:55 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636061/ [5636062] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/filestore-xfs supported-random-distro$/{centos_8} tasks/dashboard} time: 01:05:15 info: http://pulpito.ceph.com/teuthology-2020-11-18_07:01:02-rados-master-distro-basic-smithi/5636062/