2021-03-14T01:47:19.435 INFO:root:teuthology version: 1.1.0-b9656917
2021-03-14T01:47:19.437 INFO:teuthology.results:subset: '111/120000'
2021-03-14T01:47:19.438 INFO:teuthology.results:seed: '4002'
2021-03-14T05:07:26.919 INFO:root:teuthology version: 1.1.0-b9656917
2021-03-14T05:07:26.971 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-03-14T15:56:24.849 INFO:teuthology.results:Tests finished! gathering results...
2021-03-14T15:57:00.203 INFO:scrape:Found 320 jobs
2021-03-14T15:57:01.876 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 15.2.9-141-g381387f1 (381387f151822ee4d4927f3fc584ff1af9deb9ce) octopus (stable)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x154) [0x7f4b60d491a0]
,  2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f4b60d4937b]

2021-03-14T15:57:01.878 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T04:57:29.681 INFO:teuthology.orchestra.run.smithi031.stdout:
, dumped all
, 2021-03-14T04:57:29.695 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2021-03-14T04:57:29.378625+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30221142708,"num_objects":10172,"num_object_clones":0,"num_object_copies":20344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10144,"num_whiteouts":40,"num_read":22845,"num_read_kb":1501895,"num_write":53240,"num_write_kb":24110079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":13882,"num_flush":3617,"num_flush_kb":10535558,"num_evict":3241,"num_evict_kb":9431050,"num_promote":4012,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49216,"ondisk_log_size":49216,"up":32,"acting":32,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":32,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":40740996,"kb_used_data":521585828,"kb_used_omap":74122,"kb_used_meta":0,"kb_avail":521455284,"statfs":{"total":575688990720,"available":533970210816,"internally_reserved":133677056,"allocated":534103887872,"data_stored":534103887872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":75901399,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":17,"commit_latency_ns":9000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":279044344,"num_objects":118,"num_object_clones":0,"num_object_copies":236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":21,"num_read":1073,"num_read_kb":969215,"num_write":653,"num_write_kb":525513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":995,"num_flush":84,"num_flush_kb":218105,"num_evict":55,"num_evict_kb":133409,"num_promote":133,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002931"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"27","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.487498+0000","last_change":"2021-03-14T04:54:12.642936+0000","last_active":"2021-03-14T04:56:58.487498+0000","last_peered":"2021-03-14T04:56:58.487498+0000","last_clean":"2021-03-14T04:56:58.487498+0000","last_became_active":"2021-03-14T04:54:12.642317+0000","last_became_peered":"2021-03-14T04:54:12.642317+0000","last_unstale":"2021-03-14T04:56:58.487498+0000","last_undegraded":"2021-03-14T04:56:58.487498+0000","last_fullsized":"2021-03-14T04:56:58.487498+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"26","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.487087+0000","last_change":"2021-03-14T04:54:12.418031+0000","last_active":"2021-03-14T04:56:58.487087+0000","last_peered":"2021-03-14T04:56:58.487087+0000","last_clean":"2021-03-14T04:56:58.487087+0000","last_became_active":"2021-03-14T04:54:12.417412+0000","last_became_peered":"2021-03-14T04:54:12.417412+0000","last_unstale":"2021-03-14T04:56:58.487087+0000","last_undegraded":"2021-03-14T04:56:58.487087+0000","last_fullsized":"2021-03-14T04:56:58.487087+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"32'5843","reported_seq":"8078","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:57:23.366193+0000","last_change":"2021-03-14T04:54:23.443564+0000","last_active":"2021-03-14T04:57:23.366193+0000","last_peered":"2021-03-14T04:57:23.366193+0000","last_clean":"2021-03-14T04:57:23.366193+0000","last_became_active":"2021-03-14T04:54:23.443237+0000","last_became_peered":"2021-03-14T04:54:23.443237+0000","last_unstale":"2021-03-14T04:57:23.366193+0000","last_undegraded":"2021-03-14T04:57:23.366193+0000","last_fullsized":"2021-03-14T04:57:23.366193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_clean_scrub_stamp":"2021-03-14T04:54:22.413390+0000","log_size":5843,"ondisk_log_size":5843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":138267928,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":11,"num_read":1974,"num_read_kb":990,"num_write":5143,"num_write_kb":1198886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":3937,"num_flush":923,"num_flush_kb":2682369,"num_evict":856,"num_evict_kb":2480754,"num_promote":1047,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"33'6710","reported_seq":"8801","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:24.300664+0000","last_change":"2021-03-14T04:54:21.535219+0000","last_active":"2021-03-14T04:57:24.300664+0000","last_peered":"2021-03-14T04:57:24.300664+0000","last_clean":"2021-03-14T04:57:24.300664+0000","last_became_active":"2021-03-14T04:54:21.534448+0000","last_became_peered":"2021-03-14T04:54:21.534448+0000","last_unstale":"2021-03-14T04:57:24.300664+0000","last_undegraded":"2021-03-14T04:57:24.300664+0000","last_fullsized":"2021-03-14T04:57:24.300664+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_clean_scrub_stamp":"2021-03-14T04:54:20.170538+0000","log_size":6710,"ondisk_log_size":6710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7433652118,"num_objects":2487,"num_object_clones":0,"num_object_copies":4974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2487,"num_whiteouts":0,"num_read":3750,"num_read_kb":341397,"num_write":8348,"num_write_kb":4729158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"27","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.486920+0000","last_change":"2021-03-14T04:54:12.419664+0000","last_active":"2021-03-14T04:56:58.486920+0000","last_peered":"2021-03-14T04:56:58.486920+0000","last_clean":"2021-03-14T04:56:58.486920+0000","last_became_active":"2021-03-14T04:54:12.418921+0000","last_became_peered":"2021-03-14T04:54:12.418921+0000","last_unstale":"2021-03-14T04:56:58.486920+0000","last_undegraded":"2021-03-14T04:56:58.486920+0000","last_fullsized":"2021-03-14T04:56:58.486920+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"33'5405","reported_seq":"7509","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:24.302411+0000","last_change":"2021-03-14T04:54:23.478628+0000","last_active":"2021-03-14T04:57:24.302411+0000","last_peered":"2021-03-14T04:57:24.302411+0000","last_clean":"2021-03-14T04:57:24.302411+0000","last_became_active":"2021-03-14T04:54:23.477906+0000","last_became_peered":"2021-03-14T04:54:23.477906+0000","last_unstale":"2021-03-14T04:57:24.302411+0000","last_undegraded":"2021-03-14T04:57:24.302411+0000","last_fullsized":"2021-03-14T04:57:24.302411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_clean_scrub_stamp":"2021-03-14T04:54:22.413390+0000","log_size":5405,"ondisk_log_size":5405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":109356376,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":9,"num_read":1885,"num_read_kb":916,"num_write":4890,"num_write_kb":1177491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":3022,"num_flush":870,"num_flush_kb":2517847,"num_evict":799,"num_evict_kb":2306754,"num_promote":955,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"33'6487","reported_seq":"8657","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:23.833806+0000","last_change":"2021-03-14T04:54:21.421319+0000","last_active":"2021-03-14T04:57:23.833806+0000","last_peered":"2021-03-14T04:57:23.833806+0000","last_clean":"2021-03-14T04:57:23.833806+0000","last_became_active":"2021-03-14T04:54:21.420795+0000","last_became_peered":"2021-03-14T04:54:21.420795+0000","last_unstale":"2021-03-14T04:57:23.833806+0000","last_undegraded":"2021-03-14T04:57:23.833806+0000","last_fullsized":"2021-03-14T04:57:23.833806+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_clean_scrub_stamp":"2021-03-14T04:54:20.170538+0000","log_size":6487,"ondisk_log_size":6487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7305303989,"num_objects":2443,"num_object_clones":0,"num_object_copies":4886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2443,"num_whiteouts":0,"num_read":3657,"num_read_kb":297318,"num_write":8017,"num_write_kb":4824114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"26","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.487176+0000","last_change":"2021-03-14T04:54:12.418478+0000","last_active":"2021-03-14T04:56:58.487176+0000","last_peered":"2021-03-14T04:56:58.487176+0000","last_clean":"2021-03-14T04:56:58.487176+0000","last_became_active":"2021-03-14T04:54:12.417271+0000","last_became_peered":"2021-03-14T04:54:12.417271+0000","last_unstale":"2021-03-14T04:56:58.487176+0000","last_undegraded":"2021-03-14T04:56:58.487176+0000","last_fullsized":"2021-03-14T04:56:58.487176+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"33'5472","reported_seq":"7685","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:24.757059+0000","last_change":"2021-03-14T04:54:23.645513+0000","last_active":"2021-03-14T04:57:24.757059+0000","last_peered":"2021-03-14T04:57:24.757059+0000","last_clean":"2021-03-14T04:57:24.757059+0000","last_became_active":"2021-03-14T04:54:23.645209+0000","last_became_peered":"2021-03-14T04:54:23.645209+0000","last_unstale":"2021-03-14T04:57:24.757059+0000","last_undegraded":"2021-03-14T04:57:24.757059+0000","last_fullsized":"2021-03-14T04:57:24.757059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_clean_scrub_stamp":"2021-03-14T04:54:22.413390+0000","log_size":5472,"ondisk_log_size":5472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":114590463,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":9,"num_read":1957,"num_read_kb":918,"num_write":4911,"num_write_kb":1177568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":3004,"num_flush":880,"num_flush_kb":2550172,"num_evict":730,"num_evict_kb":2118554,"num_promote":958,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"33'6800","reported_seq":"9068","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:23.963160+0000","last_change":"2021-03-14T04:54:21.532921+0000","last_active":"2021-03-14T04:57:23.963160+0000","last_peered":"2021-03-14T04:57:23.963160+0000","last_clean":"2021-03-14T04:57:23.963160+0000","last_became_active":"2021-03-14T04:54:21.532542+0000","last_became_peered":"2021-03-14T04:54:21.532542+0000","last_unstale":"2021-03-14T04:57:23.963160+0000","last_undegraded":"2021-03-14T04:57:23.963160+0000","last_fullsized":"2021-03-14T04:57:23.963160+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_clean_scrub_stamp":"2021-03-14T04:54:20.170538+0000","log_size":6800,"ondisk_log_size":6800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7679500993,"num_objects":2557,"num_object_clones":0,"num_object_copies":5114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2557,"num_whiteouts":0,"num_read":3925,"num_read_kb":429442,"num_write":8424,"num_write_kb":5020889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"27","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.486867+0000","last_change":"2021-03-14T04:54:12.418875+0000","last_active":"2021-03-14T04:56:58.486867+0000","last_peered":"2021-03-14T04:56:58.486867+0000","last_clean":"2021-03-14T04:56:58.486867+0000","last_became_active":"2021-03-14T04:54:12.418565+0000","last_became_peered":"2021-03-14T04:54:12.418565+0000","last_unstale":"2021-03-14T04:56:58.486867+0000","last_undegraded":"2021-03-14T04:56:58.486867+0000","last_fullsized":"2021-03-14T04:56:58.486867+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"32'5870","reported_seq":"8152","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:57:22.983929+0000","last_change":"2021-03-14T04:54:23.442763+0000","last_active":"2021-03-14T04:57:22.983929+0000","last_peered":"2021-03-14T04:57:22.983929+0000","last_clean":"2021-03-14T04:57:22.983929+0000","last_became_active":"2021-03-14T04:54:23.442127+0000","last_became_peered":"2021-03-14T04:54:23.442127+0000","last_unstale":"2021-03-14T04:57:22.983929+0000","last_undegraded":"2021-03-14T04:57:22.983929+0000","last_fullsized":"2021-03-14T04:57:22.983929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_clean_scrub_stamp":"2021-03-14T04:54:22.413390+0000","log_size":5870,"ondisk_log_size":5870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":123608883,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":11,"num_read":2007,"num_read_kb":998,"num_write":5269,"num_write_kb":1250136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":3919,"num_flush":944,"num_flush_kb":2785170,"num_evict":856,"num_evict_kb":2524988,"num_promote":1052,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"33'6629","reported_seq":"8659","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:24.752363+0000","last_change":"2021-03-14T04:54:21.534780+0000","last_active":"2021-03-14T04:57:24.752363+0000","last_peered":"2021-03-14T04:57:24.752363+0000","last_clean":"2021-03-14T04:57:24.752363+0000","last_became_active":"2021-03-14T04:54:21.533809+0000","last_became_peered":"2021-03-14T04:54:21.533809+0000","last_unstale":"2021-03-14T04:57:24.752363+0000","last_undegraded":"2021-03-14T04:57:24.752363+0000","last_fullsized":"2021-03-14T04:57:24.752363+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_clean_scrub_stamp":"2021-03-14T04:54:20.170538+0000","log_size":6629,"ondisk_log_size":6629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7316861958,"num_objects":2453,"num_object_clones":0,"num_object_copies":4906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2453,"num_whiteouts":0,"num_read":3690,"num_read_kb":429916,"num_write":8238,"num_write_kb":4731837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"26","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.484680+0000","last_change":"2021-03-14T04:54:12.195854+0000","last_active":"2021-03-14T04:56:58.484680+0000","last_peered":"2021-03-14T04:56:58.484680+0000","last_clean":"2021-03-14T04:56:58.484680+0000","last_became_active":"2021-03-14T04:54:12.195060+0000","last_became_peered":"2021-03-14T04:54:12.195060+0000","last_unstale":"2021-03-14T04:56:58.484680+0000","last_undegraded":"2021-03-14T04:56:58.484680+0000","last_fullsized":"2021-03-14T04:56:58.484680+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"26","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.484919+0000","last_change":"2021-03-14T04:54:12.195721+0000","last_active":"2021-03-14T04:56:58.484919+0000","last_peered":"2021-03-14T04:56:58.484919+0000","last_clean":"2021-03-14T04:56:58.484919+0000","last_became_active":"2021-03-14T04:54:12.194809+0000","last_became_peered":"2021-03-14T04:54:12.194809+0000","last_unstale":"2021-03-14T04:56:58.484919+0000","last_undegraded":"2021-03-14T04:56:58.484919+0000","last_fullsized":"2021-03-14T04:56:58.484919+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"27","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.486886+0000","last_change":"2021-03-14T04:54:12.643927+0000","last_active":"2021-03-14T04:56:58.486886+0000","last_peered":"2021-03-14T04:56:58.486886+0000","last_clean":"2021-03-14T04:56:58.486886+0000","last_became_active":"2021-03-14T04:54:12.643112+0000","last_became_peered":"2021-03-14T04:54:12.643112+0000","last_unstale":"2021-03-14T04:56:58.486886+0000","last_undegraded":"2021-03-14T04:56:58.486886+0000","last_fullsized":"2021-03-14T04:56:58.486886+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":485823650,"num_objects":232,"num_object_clones":0,"num_object_copies":464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":204,"num_whiteouts":40,"num_read":7823,"num_read_kb":3822,"num_write":20213,"num_write_kb":4804081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":13882,"num_flush":3617,"num_flush_kb":10535558,"num_evict":3241,"num_evict_kb":9431050,"num_promote":4012,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22590,"ondisk_log_size":22590,"up":8,"acting":8,"num_store_stats":0},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":2,"num_pg":4,"stat_sum":{"num_bytes":29735319058,"num_objects":9940,"num_object_clones":0,"num_object_copies":19880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9940,"num_whiteouts":0,"num_read":15022,"num_read_kb":1498073,"num_write":33027,"num_write_kb":19305998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26626,"ondisk_log_size":26626,"up":8,"acting":8,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640298,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":9873360,"kb_used_data":83864652,"kb_used_omap":12353,"kb_used_meta":0,"kb_avail":83826020,"statfs":{"total":95948165120,"available":85837844480,"internally_reserved":39559168,"allocated":85877403648,"data_stored":85877403648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12649818,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":3,"commit_latency_ns":2000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.505,"15min":0.51500000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.82499999999999996,"5min":1.171,"15min":1.171},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.437,"5min":0.49399999999999999,"15min":0.50800000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.89900000000000002}]},{"osd":1,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.54300000000000004,"15min":0.56799999999999995},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.88600000000000001,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.52300000000000002,"15min":0.54100000000000004},"min":{"1min":0.28599999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.85399999999999998,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.65300000000000002}]},{"osd":2,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.58399999999999996,"15min":0.55500000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":2.681,"5min":2.681,"15min":2.681},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.622,"15min":0.60999999999999999},"min":{"1min":0.32000000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":4.1689999999999996,"5min":4.1689999999999996,"15min":4.1689999999999996},"last":0.98699999999999999}]},{"osd":3,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.502,"15min":0.51700000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.46800000000000003,"15min":0.48599999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.80700000000000005}]},{"osd":4,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.52500000000000002,"15min":0.54500000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.89400000000000002,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.872},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.51700000000000002,"15min":0.53300000000000003},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":1.845,"5min":1.845,"15min":1.845},"last":0.83499999999999996}]}]},{"osd":4,"up_from":11,"seq":47244640299,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":488804,"kb_used_data":93226564,"kb_used_omap":12172,"kb_used_meta":0,"kb_avail":93210576,"statfs":{"total":95948165120,"available":95447629824,"internally_reserved":16371712,"allocated":95464001536,"data_stored":95464001536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12464664,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":2,"commit_latency_ns":1000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.505,"15min":0.54900000000000004},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.69399999999999995,"5min":1.036,"15min":1.036},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.53000000000000003,"15min":0.55400000000000005},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.72399999999999998}]},{"osd":1,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.9790000000000001,"5min":1.9790000000000001,"15min":1.9790000000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.53600000000000003,"15min":0.57099999999999995},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.72699999999999998,"5min":1.075,"15min":1.075},"last":0.75600000000000001}]},{"osd":2,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.52500000000000002,"15min":0.56499999999999995},"min":{"1min":0.314,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.72699999999999998,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.58299999999999996,"15min":0.629},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.91100000000000003,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.56399999999999995}]},{"osd":3,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.46400000000000002,"15min":0.50900000000000001},"min":{"1min":0.23699999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.52600000000000002,"5min":1.202,"15min":1.202},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.48799999999999999,"15min":0.52100000000000002},"min":{"1min":0.246,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.55900000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.629}]},{"osd":5,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.45100000000000001,"15min":0.47299999999999998},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.81299999999999994,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.5,"15min":0.54400000000000004},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.59699999999999998,"5min":1.121,"15min":1.121},"last":0.76500000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640299,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":4952008,"kb_used_data":88768532,"kb_used_omap":3548,"kb_used_meta":0,"kb_avail":88747372,"statfs":{"total":95948165120,"available":90877308928,"internally_reserved":21667840,"allocated":90898976768,"data_stored":90898976768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3633814,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.54700000000000004,"15min":0.59899999999999998},"min":{"1min":0.253,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.89700000000000002,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":1},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.505,"15min":0.53200000000000003},"min":{"1min":0.35299999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.91600000000000004,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":1.2270000000000001}]},{"osd":1,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.52200000000000002,"15min":0.55400000000000005},"min":{"1min":0.27400000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.77800000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.505,"5min":0.53500000000000003,"15min":0.56499999999999995},"min":{"1min":0.29299999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.93200000000000005}]},{"osd":2,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.54000000000000004,"15min":0.57799999999999996},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.68500000000000005,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.55000000000000004,"15min":0.57199999999999995},"min":{"1min":0.34999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.56399999999999995}]},{"osd":4,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.46600000000000003,"15min":0.51100000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.53300000000000003,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":1.1859999999999999},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.47099999999999997,"15min":0.52300000000000002},"min":{"1min":0.247,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.57499999999999996,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.65600000000000003}]},{"osd":5,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.46100000000000002,"15min":0.49199999999999999},"min":{"1min":0.24299999999999999,"5min":0.215,"15min":0.215},"max":{"1min":0.65600000000000003,"5min":0.874,"15min":0.874},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.53200000000000003,"15min":0.57999999999999996},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.69599999999999995,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.751}]}]},{"osd":2,"up_from":11,"seq":47244640298,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":10090296,"kb_used_data":83630492,"kb_used_omap":12460,"kb_used_meta":0,"kb_avail":83609084,"statfs":{"total":95948165120,"available":85615702016,"internally_reserved":21921792,"allocated":85637623808,"data_stored":85637623808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12759532,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":4,"commit_latency_ns":3000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.52400000000000002,"15min":0.48399999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":10.157999999999999,"5min":10.157999999999999,"15min":10.157999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.51600000000000001,"15min":0.502},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.984,"5min":5.984,"15min":5.984},"last":0.51200000000000001}]},{"osd":1,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.67200000000000004,"15min":0.627},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.2140000000000004,"5min":6.2140000000000004,"15min":6.2140000000000004},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.59299999999999997,"15min":0.56599999999999995},"min":{"1min":0.25600000000000001,"5min":0.219,"15min":0.219},"max":{"1min":3.7749999999999999,"5min":3.7749999999999999,"15min":3.7749999999999999},"last":0.49199999999999999}]},{"osd":3,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.70199999999999996,"15min":0.69499999999999995},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":6.2309999999999999,"5min":6.2309999999999999,"15min":6.2309999999999999},"last":0.434},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.71699999999999997,"15min":0.71899999999999997},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":6.0199999999999996,"5min":6.0199999999999996,"15min":6.0199999999999996},"last":0.45500000000000002}]},{"osd":4,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":6.3499999999999996,"5min":6.3499999999999996,"15min":6.3499999999999996},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.68899999999999995,"15min":0.66900000000000004},"min":{"1min":0.379,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":6.3040000000000003,"5min":6.3040000000000003,"15min":6.3040000000000003},"last":0.52200000000000002}]},{"osd":5,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.72099999999999997,"15min":0.72299999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":6.0490000000000004,"5min":6.0490000000000004,"15min":6.0490000000000004},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.69599999999999995,"15min":0.72999999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":0.70099999999999996}]}]},{"osd":0,"up_from":11,"seq":47244640298,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":498748,"kb_used_data":93219896,"kb_used_omap":12182,"kb_used_meta":0,"kb_avail":93200632,"statfs":{"total":95948165120,"available":95437447168,"internally_reserved":19726336,"allocated":95457173504,"data_stored":95457173504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12474636,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":3,"commit_latency_ns":1000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.44900000000000001,"15min":0.45900000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.503},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.53400000000000003,"15min":0.54600000000000004},"min":{"1min":0.251,"5min":0.249,"15min":0.249},"max":{"1min":2.2509999999999999,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":0.61399999999999999}]},{"osd":2,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56499999999999995,"15min":0.59299999999999997},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.351,"5min":2.351,"15min":2.351},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.57399999999999995,"15min":0.63400000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.80900000000000005,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.67400000000000004}]},{"osd":3,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.58899999999999997,"15min":0.60699999999999998},"min":{"1min":0.31900000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":2.4769999999999999,"5min":2.4769999999999999,"15min":2.4769999999999999},"last":0.746},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.64400000000000002,"15min":0.68999999999999995},"min":{"1min":0.36499999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71399999999999997,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.71099999999999997}]},{"osd":4,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.65400000000000003,"15min":0.71699999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71999999999999997,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.70299999999999996,"15min":0.73599999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":2.4409999999999998,"5min":2.4409999999999998,"15min":2.4409999999999998},"last":0.78700000000000003}]},{"osd":5,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.68799999999999994,"15min":0.74299999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.78400000000000003,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.73199999999999998,"15min":0.80800000000000005},"min":{"1min":0.38900000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.77700000000000002,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":0.755}]}]},{"osd":1,"up_from":11,"seq":47244640298,"num_pgs":7,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":14837780,"kb_used_data":78875692,"kb_used_omap":21405,"kb_used_meta":0,"kb_avail":78861600,"statfs":{"total":95948165120,"available":80754278400,"internally_reserved":14430208,"allocated":80768708608,"data_stored":80768708608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21918935,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":3,"commit_latency_ns":2000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.47099999999999997,"15min":0.47999999999999998},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":1.0609999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.308},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.56100000000000005,"15min":0.61299999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.806,"5min":1.806,"15min":1.806},"last":0.34300000000000003}]},{"osd":2,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.501,"15min":0.52800000000000002},"min":{"1min":0.24299999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.60099999999999998,"15min":0.61799999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":2.2570000000000001,"5min":2.2570000000000001,"15min":2.2570000000000001},"last":0.35899999999999999}]},{"osd":3,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.63,"15min":0.69499999999999995},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.0740000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.499},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.621,"15min":0.65500000000000003},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.069,"5min":1.127,"15min":1.127},"last":0.60199999999999998}]},{"osd":4,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.68000000000000005,"15min":0.70399999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.68200000000000005,"15min":0.73799999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.073,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.745}]},{"osd":5,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.68899999999999995,"15min":0.74099999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.73899999999999999,"15min":0.73499999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.1669999999999998,"5min":4.1669999999999998,"15min":4.1669999999999998},"last":0.61499999999999999}]}]}]}}

2021-03-14T15:57:01.880 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T04:58:51.516 INFO:teuthology.orchestra.run:waiting for 300
, 2021-03-14T04:58:51.575 INFO:tasks.ceph.mon.a:Stopped
, 2021-03-14T04:58:51.575 INFO:tasks.ceph:Checking cluster log for badness...

2021-03-14T15:57:01.881 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T04:59:10.571 INFO:teuthology.task.internal.syslog:Checking logs for errors...
, 2021-03-14T04:59:10.572 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com
, 2021-03-14T04:59:10.572 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1

2021-03-14T15:57:01.881 WARNING:scrape:Ignoring malparsed backtrace:     dump_program = dump_out.split("from '")[1].split(' ')[0]
, IndexError: list index out of range
, 2021-03-14T04:59:35.497 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload

2021-03-14T15:57:01.993 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 15.2.9-141-g381387f1 (381387f151822ee4d4927f3fc584ff1af9deb9ce) octopus (stable)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x154) [0x7f4b60d491a0]
,  2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f4b60d4937b]

2021-03-14T15:57:01.995 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T04:57:29.681 INFO:teuthology.orchestra.run.smithi031.stdout:
, dumped all
, 2021-03-14T04:57:29.695 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2021-03-14T04:57:29.378625+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30221142708,"num_objects":10172,"num_object_clones":0,"num_object_copies":20344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10144,"num_whiteouts":40,"num_read":22845,"num_read_kb":1501895,"num_write":53240,"num_write_kb":24110079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":13882,"num_flush":3617,"num_flush_kb":10535558,"num_evict":3241,"num_evict_kb":9431050,"num_promote":4012,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49216,"ondisk_log_size":49216,"up":32,"acting":32,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":32,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":40740996,"kb_used_data":521585828,"kb_used_omap":74122,"kb_used_meta":0,"kb_avail":521455284,"statfs":{"total":575688990720,"available":533970210816,"internally_reserved":133677056,"allocated":534103887872,"data_stored":534103887872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":75901399,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":17,"commit_latency_ns":9000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":279044344,"num_objects":118,"num_object_clones":0,"num_object_copies":236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":21,"num_read":1073,"num_read_kb":969215,"num_write":653,"num_write_kb":525513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":995,"num_flush":84,"num_flush_kb":218105,"num_evict":55,"num_evict_kb":133409,"num_promote":133,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002931"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"27","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.487498+0000","last_change":"2021-03-14T04:54:12.642936+0000","last_active":"2021-03-14T04:56:58.487498+0000","last_peered":"2021-03-14T04:56:58.487498+0000","last_clean":"2021-03-14T04:56:58.487498+0000","last_became_active":"2021-03-14T04:54:12.642317+0000","last_became_peered":"2021-03-14T04:54:12.642317+0000","last_unstale":"2021-03-14T04:56:58.487498+0000","last_undegraded":"2021-03-14T04:56:58.487498+0000","last_fullsized":"2021-03-14T04:56:58.487498+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"26","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.487087+0000","last_change":"2021-03-14T04:54:12.418031+0000","last_active":"2021-03-14T04:56:58.487087+0000","last_peered":"2021-03-14T04:56:58.487087+0000","last_clean":"2021-03-14T04:56:58.487087+0000","last_became_active":"2021-03-14T04:54:12.417412+0000","last_became_peered":"2021-03-14T04:54:12.417412+0000","last_unstale":"2021-03-14T04:56:58.487087+0000","last_undegraded":"2021-03-14T04:56:58.487087+0000","last_fullsized":"2021-03-14T04:56:58.487087+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"32'5843","reported_seq":"8078","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:57:23.366193+0000","last_change":"2021-03-14T04:54:23.443564+0000","last_active":"2021-03-14T04:57:23.366193+0000","last_peered":"2021-03-14T04:57:23.366193+0000","last_clean":"2021-03-14T04:57:23.366193+0000","last_became_active":"2021-03-14T04:54:23.443237+0000","last_became_peered":"2021-03-14T04:54:23.443237+0000","last_unstale":"2021-03-14T04:57:23.366193+0000","last_undegraded":"2021-03-14T04:57:23.366193+0000","last_fullsized":"2021-03-14T04:57:23.366193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_clean_scrub_stamp":"2021-03-14T04:54:22.413390+0000","log_size":5843,"ondisk_log_size":5843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":138267928,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":11,"num_read":1974,"num_read_kb":990,"num_write":5143,"num_write_kb":1198886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":3937,"num_flush":923,"num_flush_kb":2682369,"num_evict":856,"num_evict_kb":2480754,"num_promote":1047,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"33'6710","reported_seq":"8801","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:24.300664+0000","last_change":"2021-03-14T04:54:21.535219+0000","last_active":"2021-03-14T04:57:24.300664+0000","last_peered":"2021-03-14T04:57:24.300664+0000","last_clean":"2021-03-14T04:57:24.300664+0000","last_became_active":"2021-03-14T04:54:21.534448+0000","last_became_peered":"2021-03-14T04:54:21.534448+0000","last_unstale":"2021-03-14T04:57:24.300664+0000","last_undegraded":"2021-03-14T04:57:24.300664+0000","last_fullsized":"2021-03-14T04:57:24.300664+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_clean_scrub_stamp":"2021-03-14T04:54:20.170538+0000","log_size":6710,"ondisk_log_size":6710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7433652118,"num_objects":2487,"num_object_clones":0,"num_object_copies":4974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2487,"num_whiteouts":0,"num_read":3750,"num_read_kb":341397,"num_write":8348,"num_write_kb":4729158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"27","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.486920+0000","last_change":"2021-03-14T04:54:12.419664+0000","last_active":"2021-03-14T04:56:58.486920+0000","last_peered":"2021-03-14T04:56:58.486920+0000","last_clean":"2021-03-14T04:56:58.486920+0000","last_became_active":"2021-03-14T04:54:12.418921+0000","last_became_peered":"2021-03-14T04:54:12.418921+0000","last_unstale":"2021-03-14T04:56:58.486920+0000","last_undegraded":"2021-03-14T04:56:58.486920+0000","last_fullsized":"2021-03-14T04:56:58.486920+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"33'5405","reported_seq":"7509","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:24.302411+0000","last_change":"2021-03-14T04:54:23.478628+0000","last_active":"2021-03-14T04:57:24.302411+0000","last_peered":"2021-03-14T04:57:24.302411+0000","last_clean":"2021-03-14T04:57:24.302411+0000","last_became_active":"2021-03-14T04:54:23.477906+0000","last_became_peered":"2021-03-14T04:54:23.477906+0000","last_unstale":"2021-03-14T04:57:24.302411+0000","last_undegraded":"2021-03-14T04:57:24.302411+0000","last_fullsized":"2021-03-14T04:57:24.302411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_clean_scrub_stamp":"2021-03-14T04:54:22.413390+0000","log_size":5405,"ondisk_log_size":5405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":109356376,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":9,"num_read":1885,"num_read_kb":916,"num_write":4890,"num_write_kb":1177491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":3022,"num_flush":870,"num_flush_kb":2517847,"num_evict":799,"num_evict_kb":2306754,"num_promote":955,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"33'6487","reported_seq":"8657","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:23.833806+0000","last_change":"2021-03-14T04:54:21.421319+0000","last_active":"2021-03-14T04:57:23.833806+0000","last_peered":"2021-03-14T04:57:23.833806+0000","last_clean":"2021-03-14T04:57:23.833806+0000","last_became_active":"2021-03-14T04:54:21.420795+0000","last_became_peered":"2021-03-14T04:54:21.420795+0000","last_unstale":"2021-03-14T04:57:23.833806+0000","last_undegraded":"2021-03-14T04:57:23.833806+0000","last_fullsized":"2021-03-14T04:57:23.833806+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_clean_scrub_stamp":"2021-03-14T04:54:20.170538+0000","log_size":6487,"ondisk_log_size":6487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7305303989,"num_objects":2443,"num_object_clones":0,"num_object_copies":4886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2443,"num_whiteouts":0,"num_read":3657,"num_read_kb":297318,"num_write":8017,"num_write_kb":4824114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"26","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.487176+0000","last_change":"2021-03-14T04:54:12.418478+0000","last_active":"2021-03-14T04:56:58.487176+0000","last_peered":"2021-03-14T04:56:58.487176+0000","last_clean":"2021-03-14T04:56:58.487176+0000","last_became_active":"2021-03-14T04:54:12.417271+0000","last_became_peered":"2021-03-14T04:54:12.417271+0000","last_unstale":"2021-03-14T04:56:58.487176+0000","last_undegraded":"2021-03-14T04:56:58.487176+0000","last_fullsized":"2021-03-14T04:56:58.487176+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"33'5472","reported_seq":"7685","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:24.757059+0000","last_change":"2021-03-14T04:54:23.645513+0000","last_active":"2021-03-14T04:57:24.757059+0000","last_peered":"2021-03-14T04:57:24.757059+0000","last_clean":"2021-03-14T04:57:24.757059+0000","last_became_active":"2021-03-14T04:54:23.645209+0000","last_became_peered":"2021-03-14T04:54:23.645209+0000","last_unstale":"2021-03-14T04:57:24.757059+0000","last_undegraded":"2021-03-14T04:57:24.757059+0000","last_fullsized":"2021-03-14T04:57:24.757059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_clean_scrub_stamp":"2021-03-14T04:54:22.413390+0000","log_size":5472,"ondisk_log_size":5472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":114590463,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":9,"num_read":1957,"num_read_kb":918,"num_write":4911,"num_write_kb":1177568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":3004,"num_flush":880,"num_flush_kb":2550172,"num_evict":730,"num_evict_kb":2118554,"num_promote":958,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"33'6800","reported_seq":"9068","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:23.963160+0000","last_change":"2021-03-14T04:54:21.532921+0000","last_active":"2021-03-14T04:57:23.963160+0000","last_peered":"2021-03-14T04:57:23.963160+0000","last_clean":"2021-03-14T04:57:23.963160+0000","last_became_active":"2021-03-14T04:54:21.532542+0000","last_became_peered":"2021-03-14T04:54:21.532542+0000","last_unstale":"2021-03-14T04:57:23.963160+0000","last_undegraded":"2021-03-14T04:57:23.963160+0000","last_fullsized":"2021-03-14T04:57:23.963160+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_clean_scrub_stamp":"2021-03-14T04:54:20.170538+0000","log_size":6800,"ondisk_log_size":6800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7679500993,"num_objects":2557,"num_object_clones":0,"num_object_copies":5114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2557,"num_whiteouts":0,"num_read":3925,"num_read_kb":429442,"num_write":8424,"num_write_kb":5020889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"27","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.486867+0000","last_change":"2021-03-14T04:54:12.418875+0000","last_active":"2021-03-14T04:56:58.486867+0000","last_peered":"2021-03-14T04:56:58.486867+0000","last_clean":"2021-03-14T04:56:58.486867+0000","last_became_active":"2021-03-14T04:54:12.418565+0000","last_became_peered":"2021-03-14T04:54:12.418565+0000","last_unstale":"2021-03-14T04:56:58.486867+0000","last_undegraded":"2021-03-14T04:56:58.486867+0000","last_fullsized":"2021-03-14T04:56:58.486867+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"32'5870","reported_seq":"8152","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:57:22.983929+0000","last_change":"2021-03-14T04:54:23.442763+0000","last_active":"2021-03-14T04:57:22.983929+0000","last_peered":"2021-03-14T04:57:22.983929+0000","last_clean":"2021-03-14T04:57:22.983929+0000","last_became_active":"2021-03-14T04:54:23.442127+0000","last_became_peered":"2021-03-14T04:54:23.442127+0000","last_unstale":"2021-03-14T04:57:22.983929+0000","last_undegraded":"2021-03-14T04:57:22.983929+0000","last_fullsized":"2021-03-14T04:57:22.983929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_clean_scrub_stamp":"2021-03-14T04:54:22.413390+0000","log_size":5870,"ondisk_log_size":5870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":123608883,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":11,"num_read":2007,"num_read_kb":998,"num_write":5269,"num_write_kb":1250136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":3919,"num_flush":944,"num_flush_kb":2785170,"num_evict":856,"num_evict_kb":2524988,"num_promote":1052,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"33'6629","reported_seq":"8659","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:24.752363+0000","last_change":"2021-03-14T04:54:21.534780+0000","last_active":"2021-03-14T04:57:24.752363+0000","last_peered":"2021-03-14T04:57:24.752363+0000","last_clean":"2021-03-14T04:57:24.752363+0000","last_became_active":"2021-03-14T04:54:21.533809+0000","last_became_peered":"2021-03-14T04:54:21.533809+0000","last_unstale":"2021-03-14T04:57:24.752363+0000","last_undegraded":"2021-03-14T04:57:24.752363+0000","last_fullsized":"2021-03-14T04:57:24.752363+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_clean_scrub_stamp":"2021-03-14T04:54:20.170538+0000","log_size":6629,"ondisk_log_size":6629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7316861958,"num_objects":2453,"num_object_clones":0,"num_object_copies":4906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2453,"num_whiteouts":0,"num_read":3690,"num_read_kb":429916,"num_write":8238,"num_write_kb":4731837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"26","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.484680+0000","last_change":"2021-03-14T04:54:12.195854+0000","last_active":"2021-03-14T04:56:58.484680+0000","last_peered":"2021-03-14T04:56:58.484680+0000","last_clean":"2021-03-14T04:56:58.484680+0000","last_became_active":"2021-03-14T04:54:12.195060+0000","last_became_peered":"2021-03-14T04:54:12.195060+0000","last_unstale":"2021-03-14T04:56:58.484680+0000","last_undegraded":"2021-03-14T04:56:58.484680+0000","last_fullsized":"2021-03-14T04:56:58.484680+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"26","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.484919+0000","last_change":"2021-03-14T04:54:12.195721+0000","last_active":"2021-03-14T04:56:58.484919+0000","last_peered":"2021-03-14T04:56:58.484919+0000","last_clean":"2021-03-14T04:56:58.484919+0000","last_became_active":"2021-03-14T04:54:12.194809+0000","last_became_peered":"2021-03-14T04:54:12.194809+0000","last_unstale":"2021-03-14T04:56:58.484919+0000","last_undegraded":"2021-03-14T04:56:58.484919+0000","last_fullsized":"2021-03-14T04:56:58.484919+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"27","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.486886+0000","last_change":"2021-03-14T04:54:12.643927+0000","last_active":"2021-03-14T04:56:58.486886+0000","last_peered":"2021-03-14T04:56:58.486886+0000","last_clean":"2021-03-14T04:56:58.486886+0000","last_became_active":"2021-03-14T04:54:12.643112+0000","last_became_peered":"2021-03-14T04:54:12.643112+0000","last_unstale":"2021-03-14T04:56:58.486886+0000","last_undegraded":"2021-03-14T04:56:58.486886+0000","last_fullsized":"2021-03-14T04:56:58.486886+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":485823650,"num_objects":232,"num_object_clones":0,"num_object_copies":464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":204,"num_whiteouts":40,"num_read":7823,"num_read_kb":3822,"num_write":20213,"num_write_kb":4804081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":13882,"num_flush":3617,"num_flush_kb":10535558,"num_evict":3241,"num_evict_kb":9431050,"num_promote":4012,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22590,"ondisk_log_size":22590,"up":8,"acting":8,"num_store_stats":0},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":2,"num_pg":4,"stat_sum":{"num_bytes":29735319058,"num_objects":9940,"num_object_clones":0,"num_object_copies":19880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9940,"num_whiteouts":0,"num_read":15022,"num_read_kb":1498073,"num_write":33027,"num_write_kb":19305998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26626,"ondisk_log_size":26626,"up":8,"acting":8,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640298,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":9873360,"kb_used_data":83864652,"kb_used_omap":12353,"kb_used_meta":0,"kb_avail":83826020,"statfs":{"total":95948165120,"available":85837844480,"internally_reserved":39559168,"allocated":85877403648,"data_stored":85877403648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12649818,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":3,"commit_latency_ns":2000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.505,"15min":0.51500000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.82499999999999996,"5min":1.171,"15min":1.171},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.437,"5min":0.49399999999999999,"15min":0.50800000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.89900000000000002}]},{"osd":1,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.54300000000000004,"15min":0.56799999999999995},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.88600000000000001,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.52300000000000002,"15min":0.54100000000000004},"min":{"1min":0.28599999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.85399999999999998,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.65300000000000002}]},{"osd":2,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.58399999999999996,"15min":0.55500000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":2.681,"5min":2.681,"15min":2.681},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.622,"15min":0.60999999999999999},"min":{"1min":0.32000000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":4.1689999999999996,"5min":4.1689999999999996,"15min":4.1689999999999996},"last":0.98699999999999999}]},{"osd":3,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.502,"15min":0.51700000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.46800000000000003,"15min":0.48599999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.80700000000000005}]},{"osd":4,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.52500000000000002,"15min":0.54500000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.89400000000000002,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.872},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.51700000000000002,"15min":0.53300000000000003},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":1.845,"5min":1.845,"15min":1.845},"last":0.83499999999999996}]}]},{"osd":4,"up_from":11,"seq":47244640299,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":488804,"kb_used_data":93226564,"kb_used_omap":12172,"kb_used_meta":0,"kb_avail":93210576,"statfs":{"total":95948165120,"available":95447629824,"internally_reserved":16371712,"allocated":95464001536,"data_stored":95464001536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12464664,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":2,"commit_latency_ns":1000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.505,"15min":0.54900000000000004},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.69399999999999995,"5min":1.036,"15min":1.036},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.53000000000000003,"15min":0.55400000000000005},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.72399999999999998}]},{"osd":1,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.9790000000000001,"5min":1.9790000000000001,"15min":1.9790000000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.53600000000000003,"15min":0.57099999999999995},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.72699999999999998,"5min":1.075,"15min":1.075},"last":0.75600000000000001}]},{"osd":2,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.52500000000000002,"15min":0.56499999999999995},"min":{"1min":0.314,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.72699999999999998,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.58299999999999996,"15min":0.629},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.91100000000000003,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.56399999999999995}]},{"osd":3,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.46400000000000002,"15min":0.50900000000000001},"min":{"1min":0.23699999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.52600000000000002,"5min":1.202,"15min":1.202},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.48799999999999999,"15min":0.52100000000000002},"min":{"1min":0.246,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.55900000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.629}]},{"osd":5,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.45100000000000001,"15min":0.47299999999999998},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.81299999999999994,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.5,"15min":0.54400000000000004},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.59699999999999998,"5min":1.121,"15min":1.121},"last":0.76500000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640299,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":4952008,"kb_used_data":88768532,"kb_used_omap":3548,"kb_used_meta":0,"kb_avail":88747372,"statfs":{"total":95948165120,"available":90877308928,"internally_reserved":21667840,"allocated":90898976768,"data_stored":90898976768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3633814,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.54700000000000004,"15min":0.59899999999999998},"min":{"1min":0.253,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.89700000000000002,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":1},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.505,"15min":0.53200000000000003},"min":{"1min":0.35299999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.91600000000000004,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":1.2270000000000001}]},{"osd":1,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.52200000000000002,"15min":0.55400000000000005},"min":{"1min":0.27400000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.77800000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.505,"5min":0.53500000000000003,"15min":0.56499999999999995},"min":{"1min":0.29299999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.93200000000000005}]},{"osd":2,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.54000000000000004,"15min":0.57799999999999996},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.68500000000000005,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.55000000000000004,"15min":0.57199999999999995},"min":{"1min":0.34999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.56399999999999995}]},{"osd":4,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.46600000000000003,"15min":0.51100000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.53300000000000003,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":1.1859999999999999},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.47099999999999997,"15min":0.52300000000000002},"min":{"1min":0.247,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.57499999999999996,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.65600000000000003}]},{"osd":5,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.46100000000000002,"15min":0.49199999999999999},"min":{"1min":0.24299999999999999,"5min":0.215,"15min":0.215},"max":{"1min":0.65600000000000003,"5min":0.874,"15min":0.874},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.53200000000000003,"15min":0.57999999999999996},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.69599999999999995,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.751}]}]},{"osd":2,"up_from":11,"seq":47244640298,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":10090296,"kb_used_data":83630492,"kb_used_omap":12460,"kb_used_meta":0,"kb_avail":83609084,"statfs":{"total":95948165120,"available":85615702016,"internally_reserved":21921792,"allocated":85637623808,"data_stored":85637623808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12759532,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":4,"commit_latency_ns":3000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.52400000000000002,"15min":0.48399999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":10.157999999999999,"5min":10.157999999999999,"15min":10.157999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.51600000000000001,"15min":0.502},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.984,"5min":5.984,"15min":5.984},"last":0.51200000000000001}]},{"osd":1,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.67200000000000004,"15min":0.627},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.2140000000000004,"5min":6.2140000000000004,"15min":6.2140000000000004},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.59299999999999997,"15min":0.56599999999999995},"min":{"1min":0.25600000000000001,"5min":0.219,"15min":0.219},"max":{"1min":3.7749999999999999,"5min":3.7749999999999999,"15min":3.7749999999999999},"last":0.49199999999999999}]},{"osd":3,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.70199999999999996,"15min":0.69499999999999995},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":6.2309999999999999,"5min":6.2309999999999999,"15min":6.2309999999999999},"last":0.434},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.71699999999999997,"15min":0.71899999999999997},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":6.0199999999999996,"5min":6.0199999999999996,"15min":6.0199999999999996},"last":0.45500000000000002}]},{"osd":4,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":6.3499999999999996,"5min":6.3499999999999996,"15min":6.3499999999999996},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.68899999999999995,"15min":0.66900000000000004},"min":{"1min":0.379,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":6.3040000000000003,"5min":6.3040000000000003,"15min":6.3040000000000003},"last":0.52200000000000002}]},{"osd":5,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.72099999999999997,"15min":0.72299999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":6.0490000000000004,"5min":6.0490000000000004,"15min":6.0490000000000004},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.69599999999999995,"15min":0.72999999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":0.70099999999999996}]}]},{"osd":0,"up_from":11,"seq":47244640298,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":498748,"kb_used_data":93219896,"kb_used_omap":12182,"kb_used_meta":0,"kb_avail":93200632,"statfs":{"total":95948165120,"available":95437447168,"internally_reserved":19726336,"allocated":95457173504,"data_stored":95457173504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12474636,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":3,"commit_latency_ns":1000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.44900000000000001,"15min":0.45900000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.503},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.53400000000000003,"15min":0.54600000000000004},"min":{"1min":0.251,"5min":0.249,"15min":0.249},"max":{"1min":2.2509999999999999,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":0.61399999999999999}]},{"osd":2,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56499999999999995,"15min":0.59299999999999997},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.351,"5min":2.351,"15min":2.351},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.57399999999999995,"15min":0.63400000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.80900000000000005,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.67400000000000004}]},{"osd":3,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.58899999999999997,"15min":0.60699999999999998},"min":{"1min":0.31900000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":2.4769999999999999,"5min":2.4769999999999999,"15min":2.4769999999999999},"last":0.746},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.64400000000000002,"15min":0.68999999999999995},"min":{"1min":0.36499999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71399999999999997,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.71099999999999997}]},{"osd":4,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.65400000000000003,"15min":0.71699999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71999999999999997,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.70299999999999996,"15min":0.73599999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":2.4409999999999998,"5min":2.4409999999999998,"15min":2.4409999999999998},"last":0.78700000000000003}]},{"osd":5,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.68799999999999994,"15min":0.74299999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.78400000000000003,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.73199999999999998,"15min":0.80800000000000005},"min":{"1min":0.38900000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.77700000000000002,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":0.755}]}]},{"osd":1,"up_from":11,"seq":47244640298,"num_pgs":7,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":14837780,"kb_used_data":78875692,"kb_used_omap":21405,"kb_used_meta":0,"kb_avail":78861600,"statfs":{"total":95948165120,"available":80754278400,"internally_reserved":14430208,"allocated":80768708608,"data_stored":80768708608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21918935,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":3,"commit_latency_ns":2000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.47099999999999997,"15min":0.47999999999999998},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":1.0609999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.308},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.56100000000000005,"15min":0.61299999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.806,"5min":1.806,"15min":1.806},"last":0.34300000000000003}]},{"osd":2,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.501,"15min":0.52800000000000002},"min":{"1min":0.24299999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.60099999999999998,"15min":0.61799999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":2.2570000000000001,"5min":2.2570000000000001,"15min":2.2570000000000001},"last":0.35899999999999999}]},{"osd":3,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.63,"15min":0.69499999999999995},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.0740000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.499},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.621,"15min":0.65500000000000003},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.069,"5min":1.127,"15min":1.127},"last":0.60199999999999998}]},{"osd":4,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.68000000000000005,"15min":0.70399999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.68200000000000005,"15min":0.73799999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.073,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.745}]},{"osd":5,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.68899999999999995,"15min":0.74099999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.73899999999999999,"15min":0.73499999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.1669999999999998,"5min":4.1669999999999998,"15min":4.1669999999999998},"last":0.61499999999999999}]}]}]}}

2021-03-14T15:57:01.997 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T04:58:51.516 INFO:teuthology.orchestra.run:waiting for 300
, 2021-03-14T04:58:51.575 INFO:tasks.ceph.mon.a:Stopped
, 2021-03-14T04:58:51.575 INFO:tasks.ceph:Checking cluster log for badness...

2021-03-14T15:57:01.998 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T04:59:10.571 INFO:teuthology.task.internal.syslog:Checking logs for errors...
, 2021-03-14T04:59:10.572 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com
, 2021-03-14T04:59:10.572 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1

2021-03-14T15:57:01.998 WARNING:scrape:Ignoring malparsed backtrace:     dump_program = dump_out.split("from '")[1].split(' ')[0]
, IndexError: list index out of range
, 2021-03-14T04:59:35.497 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload

2021-03-14T15:57:02.105 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 15.2.9-141-g381387f1 (381387f151822ee4d4927f3fc584ff1af9deb9ce) octopus (stable)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x154) [0x7f4b60d491a0]
,  2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f4b60d4937b]

2021-03-14T15:57:02.108 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T04:57:29.681 INFO:teuthology.orchestra.run.smithi031.stdout:
, dumped all
, 2021-03-14T04:57:29.695 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2021-03-14T04:57:29.378625+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30221142708,"num_objects":10172,"num_object_clones":0,"num_object_copies":20344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10144,"num_whiteouts":40,"num_read":22845,"num_read_kb":1501895,"num_write":53240,"num_write_kb":24110079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":13882,"num_flush":3617,"num_flush_kb":10535558,"num_evict":3241,"num_evict_kb":9431050,"num_promote":4012,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49216,"ondisk_log_size":49216,"up":32,"acting":32,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":32,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":40740996,"kb_used_data":521585828,"kb_used_omap":74122,"kb_used_meta":0,"kb_avail":521455284,"statfs":{"total":575688990720,"available":533970210816,"internally_reserved":133677056,"allocated":534103887872,"data_stored":534103887872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":75901399,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":17,"commit_latency_ns":9000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":279044344,"num_objects":118,"num_object_clones":0,"num_object_copies":236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":21,"num_read":1073,"num_read_kb":969215,"num_write":653,"num_write_kb":525513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":995,"num_flush":84,"num_flush_kb":218105,"num_evict":55,"num_evict_kb":133409,"num_promote":133,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002931"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"27","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.487498+0000","last_change":"2021-03-14T04:54:12.642936+0000","last_active":"2021-03-14T04:56:58.487498+0000","last_peered":"2021-03-14T04:56:58.487498+0000","last_clean":"2021-03-14T04:56:58.487498+0000","last_became_active":"2021-03-14T04:54:12.642317+0000","last_became_peered":"2021-03-14T04:54:12.642317+0000","last_unstale":"2021-03-14T04:56:58.487498+0000","last_undegraded":"2021-03-14T04:56:58.487498+0000","last_fullsized":"2021-03-14T04:56:58.487498+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"26","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.487087+0000","last_change":"2021-03-14T04:54:12.418031+0000","last_active":"2021-03-14T04:56:58.487087+0000","last_peered":"2021-03-14T04:56:58.487087+0000","last_clean":"2021-03-14T04:56:58.487087+0000","last_became_active":"2021-03-14T04:54:12.417412+0000","last_became_peered":"2021-03-14T04:54:12.417412+0000","last_unstale":"2021-03-14T04:56:58.487087+0000","last_undegraded":"2021-03-14T04:56:58.487087+0000","last_fullsized":"2021-03-14T04:56:58.487087+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"32'5843","reported_seq":"8078","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:57:23.366193+0000","last_change":"2021-03-14T04:54:23.443564+0000","last_active":"2021-03-14T04:57:23.366193+0000","last_peered":"2021-03-14T04:57:23.366193+0000","last_clean":"2021-03-14T04:57:23.366193+0000","last_became_active":"2021-03-14T04:54:23.443237+0000","last_became_peered":"2021-03-14T04:54:23.443237+0000","last_unstale":"2021-03-14T04:57:23.366193+0000","last_undegraded":"2021-03-14T04:57:23.366193+0000","last_fullsized":"2021-03-14T04:57:23.366193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_clean_scrub_stamp":"2021-03-14T04:54:22.413390+0000","log_size":5843,"ondisk_log_size":5843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":138267928,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":11,"num_read":1974,"num_read_kb":990,"num_write":5143,"num_write_kb":1198886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":3937,"num_flush":923,"num_flush_kb":2682369,"num_evict":856,"num_evict_kb":2480754,"num_promote":1047,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"33'6710","reported_seq":"8801","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:24.300664+0000","last_change":"2021-03-14T04:54:21.535219+0000","last_active":"2021-03-14T04:57:24.300664+0000","last_peered":"2021-03-14T04:57:24.300664+0000","last_clean":"2021-03-14T04:57:24.300664+0000","last_became_active":"2021-03-14T04:54:21.534448+0000","last_became_peered":"2021-03-14T04:54:21.534448+0000","last_unstale":"2021-03-14T04:57:24.300664+0000","last_undegraded":"2021-03-14T04:57:24.300664+0000","last_fullsized":"2021-03-14T04:57:24.300664+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_clean_scrub_stamp":"2021-03-14T04:54:20.170538+0000","log_size":6710,"ondisk_log_size":6710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7433652118,"num_objects":2487,"num_object_clones":0,"num_object_copies":4974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2487,"num_whiteouts":0,"num_read":3750,"num_read_kb":341397,"num_write":8348,"num_write_kb":4729158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"27","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.486920+0000","last_change":"2021-03-14T04:54:12.419664+0000","last_active":"2021-03-14T04:56:58.486920+0000","last_peered":"2021-03-14T04:56:58.486920+0000","last_clean":"2021-03-14T04:56:58.486920+0000","last_became_active":"2021-03-14T04:54:12.418921+0000","last_became_peered":"2021-03-14T04:54:12.418921+0000","last_unstale":"2021-03-14T04:56:58.486920+0000","last_undegraded":"2021-03-14T04:56:58.486920+0000","last_fullsized":"2021-03-14T04:56:58.486920+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"33'5405","reported_seq":"7509","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:24.302411+0000","last_change":"2021-03-14T04:54:23.478628+0000","last_active":"2021-03-14T04:57:24.302411+0000","last_peered":"2021-03-14T04:57:24.302411+0000","last_clean":"2021-03-14T04:57:24.302411+0000","last_became_active":"2021-03-14T04:54:23.477906+0000","last_became_peered":"2021-03-14T04:54:23.477906+0000","last_unstale":"2021-03-14T04:57:24.302411+0000","last_undegraded":"2021-03-14T04:57:24.302411+0000","last_fullsized":"2021-03-14T04:57:24.302411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_clean_scrub_stamp":"2021-03-14T04:54:22.413390+0000","log_size":5405,"ondisk_log_size":5405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":109356376,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":9,"num_read":1885,"num_read_kb":916,"num_write":4890,"num_write_kb":1177491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":3022,"num_flush":870,"num_flush_kb":2517847,"num_evict":799,"num_evict_kb":2306754,"num_promote":955,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"33'6487","reported_seq":"8657","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:23.833806+0000","last_change":"2021-03-14T04:54:21.421319+0000","last_active":"2021-03-14T04:57:23.833806+0000","last_peered":"2021-03-14T04:57:23.833806+0000","last_clean":"2021-03-14T04:57:23.833806+0000","last_became_active":"2021-03-14T04:54:21.420795+0000","last_became_peered":"2021-03-14T04:54:21.420795+0000","last_unstale":"2021-03-14T04:57:23.833806+0000","last_undegraded":"2021-03-14T04:57:23.833806+0000","last_fullsized":"2021-03-14T04:57:23.833806+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_clean_scrub_stamp":"2021-03-14T04:54:20.170538+0000","log_size":6487,"ondisk_log_size":6487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7305303989,"num_objects":2443,"num_object_clones":0,"num_object_copies":4886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2443,"num_whiteouts":0,"num_read":3657,"num_read_kb":297318,"num_write":8017,"num_write_kb":4824114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"26","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.487176+0000","last_change":"2021-03-14T04:54:12.418478+0000","last_active":"2021-03-14T04:56:58.487176+0000","last_peered":"2021-03-14T04:56:58.487176+0000","last_clean":"2021-03-14T04:56:58.487176+0000","last_became_active":"2021-03-14T04:54:12.417271+0000","last_became_peered":"2021-03-14T04:54:12.417271+0000","last_unstale":"2021-03-14T04:56:58.487176+0000","last_undegraded":"2021-03-14T04:56:58.487176+0000","last_fullsized":"2021-03-14T04:56:58.487176+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"33'5472","reported_seq":"7685","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:24.757059+0000","last_change":"2021-03-14T04:54:23.645513+0000","last_active":"2021-03-14T04:57:24.757059+0000","last_peered":"2021-03-14T04:57:24.757059+0000","last_clean":"2021-03-14T04:57:24.757059+0000","last_became_active":"2021-03-14T04:54:23.645209+0000","last_became_peered":"2021-03-14T04:54:23.645209+0000","last_unstale":"2021-03-14T04:57:24.757059+0000","last_undegraded":"2021-03-14T04:57:24.757059+0000","last_fullsized":"2021-03-14T04:57:24.757059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_clean_scrub_stamp":"2021-03-14T04:54:22.413390+0000","log_size":5472,"ondisk_log_size":5472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":114590463,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":9,"num_read":1957,"num_read_kb":918,"num_write":4911,"num_write_kb":1177568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":3004,"num_flush":880,"num_flush_kb":2550172,"num_evict":730,"num_evict_kb":2118554,"num_promote":958,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"33'6800","reported_seq":"9068","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:23.963160+0000","last_change":"2021-03-14T04:54:21.532921+0000","last_active":"2021-03-14T04:57:23.963160+0000","last_peered":"2021-03-14T04:57:23.963160+0000","last_clean":"2021-03-14T04:57:23.963160+0000","last_became_active":"2021-03-14T04:54:21.532542+0000","last_became_peered":"2021-03-14T04:54:21.532542+0000","last_unstale":"2021-03-14T04:57:23.963160+0000","last_undegraded":"2021-03-14T04:57:23.963160+0000","last_fullsized":"2021-03-14T04:57:23.963160+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_clean_scrub_stamp":"2021-03-14T04:54:20.170538+0000","log_size":6800,"ondisk_log_size":6800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7679500993,"num_objects":2557,"num_object_clones":0,"num_object_copies":5114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2557,"num_whiteouts":0,"num_read":3925,"num_read_kb":429442,"num_write":8424,"num_write_kb":5020889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"27","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.486867+0000","last_change":"2021-03-14T04:54:12.418875+0000","last_active":"2021-03-14T04:56:58.486867+0000","last_peered":"2021-03-14T04:56:58.486867+0000","last_clean":"2021-03-14T04:56:58.486867+0000","last_became_active":"2021-03-14T04:54:12.418565+0000","last_became_peered":"2021-03-14T04:54:12.418565+0000","last_unstale":"2021-03-14T04:56:58.486867+0000","last_undegraded":"2021-03-14T04:56:58.486867+0000","last_fullsized":"2021-03-14T04:56:58.486867+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"32'5870","reported_seq":"8152","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:57:22.983929+0000","last_change":"2021-03-14T04:54:23.442763+0000","last_active":"2021-03-14T04:57:22.983929+0000","last_peered":"2021-03-14T04:57:22.983929+0000","last_clean":"2021-03-14T04:57:22.983929+0000","last_became_active":"2021-03-14T04:54:23.442127+0000","last_became_peered":"2021-03-14T04:54:23.442127+0000","last_unstale":"2021-03-14T04:57:22.983929+0000","last_undegraded":"2021-03-14T04:57:22.983929+0000","last_fullsized":"2021-03-14T04:57:22.983929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_clean_scrub_stamp":"2021-03-14T04:54:22.413390+0000","log_size":5870,"ondisk_log_size":5870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":123608883,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":11,"num_read":2007,"num_read_kb":998,"num_write":5269,"num_write_kb":1250136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":3919,"num_flush":944,"num_flush_kb":2785170,"num_evict":856,"num_evict_kb":2524988,"num_promote":1052,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"33'6629","reported_seq":"8659","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:24.752363+0000","last_change":"2021-03-14T04:54:21.534780+0000","last_active":"2021-03-14T04:57:24.752363+0000","last_peered":"2021-03-14T04:57:24.752363+0000","last_clean":"2021-03-14T04:57:24.752363+0000","last_became_active":"2021-03-14T04:54:21.533809+0000","last_became_peered":"2021-03-14T04:54:21.533809+0000","last_unstale":"2021-03-14T04:57:24.752363+0000","last_undegraded":"2021-03-14T04:57:24.752363+0000","last_fullsized":"2021-03-14T04:57:24.752363+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_clean_scrub_stamp":"2021-03-14T04:54:20.170538+0000","log_size":6629,"ondisk_log_size":6629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7316861958,"num_objects":2453,"num_object_clones":0,"num_object_copies":4906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2453,"num_whiteouts":0,"num_read":3690,"num_read_kb":429916,"num_write":8238,"num_write_kb":4731837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"26","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.484680+0000","last_change":"2021-03-14T04:54:12.195854+0000","last_active":"2021-03-14T04:56:58.484680+0000","last_peered":"2021-03-14T04:56:58.484680+0000","last_clean":"2021-03-14T04:56:58.484680+0000","last_became_active":"2021-03-14T04:54:12.195060+0000","last_became_peered":"2021-03-14T04:54:12.195060+0000","last_unstale":"2021-03-14T04:56:58.484680+0000","last_undegraded":"2021-03-14T04:56:58.484680+0000","last_fullsized":"2021-03-14T04:56:58.484680+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"26","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.484919+0000","last_change":"2021-03-14T04:54:12.195721+0000","last_active":"2021-03-14T04:56:58.484919+0000","last_peered":"2021-03-14T04:56:58.484919+0000","last_clean":"2021-03-14T04:56:58.484919+0000","last_became_active":"2021-03-14T04:54:12.194809+0000","last_became_peered":"2021-03-14T04:54:12.194809+0000","last_unstale":"2021-03-14T04:56:58.484919+0000","last_undegraded":"2021-03-14T04:56:58.484919+0000","last_fullsized":"2021-03-14T04:56:58.484919+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"27","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.486886+0000","last_change":"2021-03-14T04:54:12.643927+0000","last_active":"2021-03-14T04:56:58.486886+0000","last_peered":"2021-03-14T04:56:58.486886+0000","last_clean":"2021-03-14T04:56:58.486886+0000","last_became_active":"2021-03-14T04:54:12.643112+0000","last_became_peered":"2021-03-14T04:54:12.643112+0000","last_unstale":"2021-03-14T04:56:58.486886+0000","last_undegraded":"2021-03-14T04:56:58.486886+0000","last_fullsized":"2021-03-14T04:56:58.486886+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":485823650,"num_objects":232,"num_object_clones":0,"num_object_copies":464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":204,"num_whiteouts":40,"num_read":7823,"num_read_kb":3822,"num_write":20213,"num_write_kb":4804081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":13882,"num_flush":3617,"num_flush_kb":10535558,"num_evict":3241,"num_evict_kb":9431050,"num_promote":4012,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22590,"ondisk_log_size":22590,"up":8,"acting":8,"num_store_stats":0},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":2,"num_pg":4,"stat_sum":{"num_bytes":29735319058,"num_objects":9940,"num_object_clones":0,"num_object_copies":19880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9940,"num_whiteouts":0,"num_read":15022,"num_read_kb":1498073,"num_write":33027,"num_write_kb":19305998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26626,"ondisk_log_size":26626,"up":8,"acting":8,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640298,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":9873360,"kb_used_data":83864652,"kb_used_omap":12353,"kb_used_meta":0,"kb_avail":83826020,"statfs":{"total":95948165120,"available":85837844480,"internally_reserved":39559168,"allocated":85877403648,"data_stored":85877403648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12649818,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":3,"commit_latency_ns":2000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.505,"15min":0.51500000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.82499999999999996,"5min":1.171,"15min":1.171},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.437,"5min":0.49399999999999999,"15min":0.50800000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.89900000000000002}]},{"osd":1,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.54300000000000004,"15min":0.56799999999999995},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.88600000000000001,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.52300000000000002,"15min":0.54100000000000004},"min":{"1min":0.28599999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.85399999999999998,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.65300000000000002}]},{"osd":2,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.58399999999999996,"15min":0.55500000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":2.681,"5min":2.681,"15min":2.681},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.622,"15min":0.60999999999999999},"min":{"1min":0.32000000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":4.1689999999999996,"5min":4.1689999999999996,"15min":4.1689999999999996},"last":0.98699999999999999}]},{"osd":3,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.502,"15min":0.51700000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.46800000000000003,"15min":0.48599999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.80700000000000005}]},{"osd":4,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.52500000000000002,"15min":0.54500000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.89400000000000002,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.872},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.51700000000000002,"15min":0.53300000000000003},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":1.845,"5min":1.845,"15min":1.845},"last":0.83499999999999996}]}]},{"osd":4,"up_from":11,"seq":47244640299,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":488804,"kb_used_data":93226564,"kb_used_omap":12172,"kb_used_meta":0,"kb_avail":93210576,"statfs":{"total":95948165120,"available":95447629824,"internally_reserved":16371712,"allocated":95464001536,"data_stored":95464001536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12464664,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":2,"commit_latency_ns":1000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.505,"15min":0.54900000000000004},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.69399999999999995,"5min":1.036,"15min":1.036},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.53000000000000003,"15min":0.55400000000000005},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.72399999999999998}]},{"osd":1,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.9790000000000001,"5min":1.9790000000000001,"15min":1.9790000000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.53600000000000003,"15min":0.57099999999999995},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.72699999999999998,"5min":1.075,"15min":1.075},"last":0.75600000000000001}]},{"osd":2,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.52500000000000002,"15min":0.56499999999999995},"min":{"1min":0.314,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.72699999999999998,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.58299999999999996,"15min":0.629},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.91100000000000003,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.56399999999999995}]},{"osd":3,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.46400000000000002,"15min":0.50900000000000001},"min":{"1min":0.23699999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.52600000000000002,"5min":1.202,"15min":1.202},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.48799999999999999,"15min":0.52100000000000002},"min":{"1min":0.246,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.55900000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.629}]},{"osd":5,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.45100000000000001,"15min":0.47299999999999998},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.81299999999999994,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.5,"15min":0.54400000000000004},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.59699999999999998,"5min":1.121,"15min":1.121},"last":0.76500000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640299,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":4952008,"kb_used_data":88768532,"kb_used_omap":3548,"kb_used_meta":0,"kb_avail":88747372,"statfs":{"total":95948165120,"available":90877308928,"internally_reserved":21667840,"allocated":90898976768,"data_stored":90898976768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3633814,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.54700000000000004,"15min":0.59899999999999998},"min":{"1min":0.253,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.89700000000000002,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":1},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.505,"15min":0.53200000000000003},"min":{"1min":0.35299999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.91600000000000004,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":1.2270000000000001}]},{"osd":1,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.52200000000000002,"15min":0.55400000000000005},"min":{"1min":0.27400000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.77800000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.505,"5min":0.53500000000000003,"15min":0.56499999999999995},"min":{"1min":0.29299999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.93200000000000005}]},{"osd":2,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.54000000000000004,"15min":0.57799999999999996},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.68500000000000005,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.55000000000000004,"15min":0.57199999999999995},"min":{"1min":0.34999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.56399999999999995}]},{"osd":4,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.46600000000000003,"15min":0.51100000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.53300000000000003,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":1.1859999999999999},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.47099999999999997,"15min":0.52300000000000002},"min":{"1min":0.247,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.57499999999999996,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.65600000000000003}]},{"osd":5,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.46100000000000002,"15min":0.49199999999999999},"min":{"1min":0.24299999999999999,"5min":0.215,"15min":0.215},"max":{"1min":0.65600000000000003,"5min":0.874,"15min":0.874},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.53200000000000003,"15min":0.57999999999999996},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.69599999999999995,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.751}]}]},{"osd":2,"up_from":11,"seq":47244640298,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":10090296,"kb_used_data":83630492,"kb_used_omap":12460,"kb_used_meta":0,"kb_avail":83609084,"statfs":{"total":95948165120,"available":85615702016,"internally_reserved":21921792,"allocated":85637623808,"data_stored":85637623808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12759532,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":4,"commit_latency_ns":3000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.52400000000000002,"15min":0.48399999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":10.157999999999999,"5min":10.157999999999999,"15min":10.157999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.51600000000000001,"15min":0.502},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.984,"5min":5.984,"15min":5.984},"last":0.51200000000000001}]},{"osd":1,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.67200000000000004,"15min":0.627},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.2140000000000004,"5min":6.2140000000000004,"15min":6.2140000000000004},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.59299999999999997,"15min":0.56599999999999995},"min":{"1min":0.25600000000000001,"5min":0.219,"15min":0.219},"max":{"1min":3.7749999999999999,"5min":3.7749999999999999,"15min":3.7749999999999999},"last":0.49199999999999999}]},{"osd":3,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.70199999999999996,"15min":0.69499999999999995},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":6.2309999999999999,"5min":6.2309999999999999,"15min":6.2309999999999999},"last":0.434},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.71699999999999997,"15min":0.71899999999999997},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":6.0199999999999996,"5min":6.0199999999999996,"15min":6.0199999999999996},"last":0.45500000000000002}]},{"osd":4,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":6.3499999999999996,"5min":6.3499999999999996,"15min":6.3499999999999996},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.68899999999999995,"15min":0.66900000000000004},"min":{"1min":0.379,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":6.3040000000000003,"5min":6.3040000000000003,"15min":6.3040000000000003},"last":0.52200000000000002}]},{"osd":5,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.72099999999999997,"15min":0.72299999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":6.0490000000000004,"5min":6.0490000000000004,"15min":6.0490000000000004},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.69599999999999995,"15min":0.72999999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":0.70099999999999996}]}]},{"osd":0,"up_from":11,"seq":47244640298,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":498748,"kb_used_data":93219896,"kb_used_omap":12182,"kb_used_meta":0,"kb_avail":93200632,"statfs":{"total":95948165120,"available":95437447168,"internally_reserved":19726336,"allocated":95457173504,"data_stored":95457173504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12474636,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":3,"commit_latency_ns":1000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.44900000000000001,"15min":0.45900000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.503},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.53400000000000003,"15min":0.54600000000000004},"min":{"1min":0.251,"5min":0.249,"15min":0.249},"max":{"1min":2.2509999999999999,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":0.61399999999999999}]},{"osd":2,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56499999999999995,"15min":0.59299999999999997},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.351,"5min":2.351,"15min":2.351},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.57399999999999995,"15min":0.63400000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.80900000000000005,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.67400000000000004}]},{"osd":3,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.58899999999999997,"15min":0.60699999999999998},"min":{"1min":0.31900000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":2.4769999999999999,"5min":2.4769999999999999,"15min":2.4769999999999999},"last":0.746},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.64400000000000002,"15min":0.68999999999999995},"min":{"1min":0.36499999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71399999999999997,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.71099999999999997}]},{"osd":4,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.65400000000000003,"15min":0.71699999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71999999999999997,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.70299999999999996,"15min":0.73599999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":2.4409999999999998,"5min":2.4409999999999998,"15min":2.4409999999999998},"last":0.78700000000000003}]},{"osd":5,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.68799999999999994,"15min":0.74299999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.78400000000000003,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.73199999999999998,"15min":0.80800000000000005},"min":{"1min":0.38900000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.77700000000000002,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":0.755}]}]},{"osd":1,"up_from":11,"seq":47244640298,"num_pgs":7,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":14837780,"kb_used_data":78875692,"kb_used_omap":21405,"kb_used_meta":0,"kb_avail":78861600,"statfs":{"total":95948165120,"available":80754278400,"internally_reserved":14430208,"allocated":80768708608,"data_stored":80768708608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21918935,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":3,"commit_latency_ns":2000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.47099999999999997,"15min":0.47999999999999998},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":1.0609999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.308},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.56100000000000005,"15min":0.61299999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.806,"5min":1.806,"15min":1.806},"last":0.34300000000000003}]},{"osd":2,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.501,"15min":0.52800000000000002},"min":{"1min":0.24299999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.60099999999999998,"15min":0.61799999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":2.2570000000000001,"5min":2.2570000000000001,"15min":2.2570000000000001},"last":0.35899999999999999}]},{"osd":3,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.63,"15min":0.69499999999999995},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.0740000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.499},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.621,"15min":0.65500000000000003},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.069,"5min":1.127,"15min":1.127},"last":0.60199999999999998}]},{"osd":4,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.68000000000000005,"15min":0.70399999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.68200000000000005,"15min":0.73799999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.073,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.745}]},{"osd":5,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.68899999999999995,"15min":0.74099999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.73899999999999999,"15min":0.73499999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.1669999999999998,"5min":4.1669999999999998,"15min":4.1669999999999998},"last":0.61499999999999999}]}]}]}}

2021-03-14T15:57:02.109 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T04:58:51.516 INFO:teuthology.orchestra.run:waiting for 300
, 2021-03-14T04:58:51.575 INFO:tasks.ceph.mon.a:Stopped
, 2021-03-14T04:58:51.575 INFO:tasks.ceph:Checking cluster log for badness...

2021-03-14T15:57:02.110 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T04:59:10.571 INFO:teuthology.task.internal.syslog:Checking logs for errors...
, 2021-03-14T04:59:10.572 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com
, 2021-03-14T04:59:10.572 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1

2021-03-14T15:57:02.111 WARNING:scrape:Ignoring malparsed backtrace:     dump_program = dump_out.split("from '")[1].split(' ')[0]
, IndexError: list index out of range
, 2021-03-14T04:59:35.497 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload

2021-03-14T15:57:02.222 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 15.2.9-141-g381387f1 (381387f151822ee4d4927f3fc584ff1af9deb9ce) octopus (stable)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x154) [0x7f4b60d491a0]
,  2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f4b60d4937b]

2021-03-14T15:57:02.224 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T04:57:29.681 INFO:teuthology.orchestra.run.smithi031.stdout:
, dumped all
, 2021-03-14T04:57:29.695 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2021-03-14T04:57:29.378625+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30221142708,"num_objects":10172,"num_object_clones":0,"num_object_copies":20344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10144,"num_whiteouts":40,"num_read":22845,"num_read_kb":1501895,"num_write":53240,"num_write_kb":24110079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":13882,"num_flush":3617,"num_flush_kb":10535558,"num_evict":3241,"num_evict_kb":9431050,"num_promote":4012,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49216,"ondisk_log_size":49216,"up":32,"acting":32,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":32,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":40740996,"kb_used_data":521585828,"kb_used_omap":74122,"kb_used_meta":0,"kb_avail":521455284,"statfs":{"total":575688990720,"available":533970210816,"internally_reserved":133677056,"allocated":534103887872,"data_stored":534103887872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":75901399,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":17,"commit_latency_ns":9000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":279044344,"num_objects":118,"num_object_clones":0,"num_object_copies":236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":21,"num_read":1073,"num_read_kb":969215,"num_write":653,"num_write_kb":525513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":995,"num_flush":84,"num_flush_kb":218105,"num_evict":55,"num_evict_kb":133409,"num_promote":133,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002931"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"27","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.487498+0000","last_change":"2021-03-14T04:54:12.642936+0000","last_active":"2021-03-14T04:56:58.487498+0000","last_peered":"2021-03-14T04:56:58.487498+0000","last_clean":"2021-03-14T04:56:58.487498+0000","last_became_active":"2021-03-14T04:54:12.642317+0000","last_became_peered":"2021-03-14T04:54:12.642317+0000","last_unstale":"2021-03-14T04:56:58.487498+0000","last_undegraded":"2021-03-14T04:56:58.487498+0000","last_fullsized":"2021-03-14T04:56:58.487498+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"26","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.487087+0000","last_change":"2021-03-14T04:54:12.418031+0000","last_active":"2021-03-14T04:56:58.487087+0000","last_peered":"2021-03-14T04:56:58.487087+0000","last_clean":"2021-03-14T04:56:58.487087+0000","last_became_active":"2021-03-14T04:54:12.417412+0000","last_became_peered":"2021-03-14T04:54:12.417412+0000","last_unstale":"2021-03-14T04:56:58.487087+0000","last_undegraded":"2021-03-14T04:56:58.487087+0000","last_fullsized":"2021-03-14T04:56:58.487087+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"32'5843","reported_seq":"8078","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:57:23.366193+0000","last_change":"2021-03-14T04:54:23.443564+0000","last_active":"2021-03-14T04:57:23.366193+0000","last_peered":"2021-03-14T04:57:23.366193+0000","last_clean":"2021-03-14T04:57:23.366193+0000","last_became_active":"2021-03-14T04:54:23.443237+0000","last_became_peered":"2021-03-14T04:54:23.443237+0000","last_unstale":"2021-03-14T04:57:23.366193+0000","last_undegraded":"2021-03-14T04:57:23.366193+0000","last_fullsized":"2021-03-14T04:57:23.366193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_clean_scrub_stamp":"2021-03-14T04:54:22.413390+0000","log_size":5843,"ondisk_log_size":5843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":138267928,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":11,"num_read":1974,"num_read_kb":990,"num_write":5143,"num_write_kb":1198886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":3937,"num_flush":923,"num_flush_kb":2682369,"num_evict":856,"num_evict_kb":2480754,"num_promote":1047,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"33'6710","reported_seq":"8801","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:24.300664+0000","last_change":"2021-03-14T04:54:21.535219+0000","last_active":"2021-03-14T04:57:24.300664+0000","last_peered":"2021-03-14T04:57:24.300664+0000","last_clean":"2021-03-14T04:57:24.300664+0000","last_became_active":"2021-03-14T04:54:21.534448+0000","last_became_peered":"2021-03-14T04:54:21.534448+0000","last_unstale":"2021-03-14T04:57:24.300664+0000","last_undegraded":"2021-03-14T04:57:24.300664+0000","last_fullsized":"2021-03-14T04:57:24.300664+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_clean_scrub_stamp":"2021-03-14T04:54:20.170538+0000","log_size":6710,"ondisk_log_size":6710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7433652118,"num_objects":2487,"num_object_clones":0,"num_object_copies":4974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2487,"num_whiteouts":0,"num_read":3750,"num_read_kb":341397,"num_write":8348,"num_write_kb":4729158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"27","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.486920+0000","last_change":"2021-03-14T04:54:12.419664+0000","last_active":"2021-03-14T04:56:58.486920+0000","last_peered":"2021-03-14T04:56:58.486920+0000","last_clean":"2021-03-14T04:56:58.486920+0000","last_became_active":"2021-03-14T04:54:12.418921+0000","last_became_peered":"2021-03-14T04:54:12.418921+0000","last_unstale":"2021-03-14T04:56:58.486920+0000","last_undegraded":"2021-03-14T04:56:58.486920+0000","last_fullsized":"2021-03-14T04:56:58.486920+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"33'5405","reported_seq":"7509","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:24.302411+0000","last_change":"2021-03-14T04:54:23.478628+0000","last_active":"2021-03-14T04:57:24.302411+0000","last_peered":"2021-03-14T04:57:24.302411+0000","last_clean":"2021-03-14T04:57:24.302411+0000","last_became_active":"2021-03-14T04:54:23.477906+0000","last_became_peered":"2021-03-14T04:54:23.477906+0000","last_unstale":"2021-03-14T04:57:24.302411+0000","last_undegraded":"2021-03-14T04:57:24.302411+0000","last_fullsized":"2021-03-14T04:57:24.302411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_clean_scrub_stamp":"2021-03-14T04:54:22.413390+0000","log_size":5405,"ondisk_log_size":5405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":109356376,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":9,"num_read":1885,"num_read_kb":916,"num_write":4890,"num_write_kb":1177491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":3022,"num_flush":870,"num_flush_kb":2517847,"num_evict":799,"num_evict_kb":2306754,"num_promote":955,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"33'6487","reported_seq":"8657","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:23.833806+0000","last_change":"2021-03-14T04:54:21.421319+0000","last_active":"2021-03-14T04:57:23.833806+0000","last_peered":"2021-03-14T04:57:23.833806+0000","last_clean":"2021-03-14T04:57:23.833806+0000","last_became_active":"2021-03-14T04:54:21.420795+0000","last_became_peered":"2021-03-14T04:54:21.420795+0000","last_unstale":"2021-03-14T04:57:23.833806+0000","last_undegraded":"2021-03-14T04:57:23.833806+0000","last_fullsized":"2021-03-14T04:57:23.833806+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_clean_scrub_stamp":"2021-03-14T04:54:20.170538+0000","log_size":6487,"ondisk_log_size":6487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7305303989,"num_objects":2443,"num_object_clones":0,"num_object_copies":4886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2443,"num_whiteouts":0,"num_read":3657,"num_read_kb":297318,"num_write":8017,"num_write_kb":4824114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"26","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.487176+0000","last_change":"2021-03-14T04:54:12.418478+0000","last_active":"2021-03-14T04:56:58.487176+0000","last_peered":"2021-03-14T04:56:58.487176+0000","last_clean":"2021-03-14T04:56:58.487176+0000","last_became_active":"2021-03-14T04:54:12.417271+0000","last_became_peered":"2021-03-14T04:54:12.417271+0000","last_unstale":"2021-03-14T04:56:58.487176+0000","last_undegraded":"2021-03-14T04:56:58.487176+0000","last_fullsized":"2021-03-14T04:56:58.487176+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"33'5472","reported_seq":"7685","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:24.757059+0000","last_change":"2021-03-14T04:54:23.645513+0000","last_active":"2021-03-14T04:57:24.757059+0000","last_peered":"2021-03-14T04:57:24.757059+0000","last_clean":"2021-03-14T04:57:24.757059+0000","last_became_active":"2021-03-14T04:54:23.645209+0000","last_became_peered":"2021-03-14T04:54:23.645209+0000","last_unstale":"2021-03-14T04:57:24.757059+0000","last_undegraded":"2021-03-14T04:57:24.757059+0000","last_fullsized":"2021-03-14T04:57:24.757059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_clean_scrub_stamp":"2021-03-14T04:54:22.413390+0000","log_size":5472,"ondisk_log_size":5472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":114590463,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":9,"num_read":1957,"num_read_kb":918,"num_write":4911,"num_write_kb":1177568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":3004,"num_flush":880,"num_flush_kb":2550172,"num_evict":730,"num_evict_kb":2118554,"num_promote":958,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"33'6800","reported_seq":"9068","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:23.963160+0000","last_change":"2021-03-14T04:54:21.532921+0000","last_active":"2021-03-14T04:57:23.963160+0000","last_peered":"2021-03-14T04:57:23.963160+0000","last_clean":"2021-03-14T04:57:23.963160+0000","last_became_active":"2021-03-14T04:54:21.532542+0000","last_became_peered":"2021-03-14T04:54:21.532542+0000","last_unstale":"2021-03-14T04:57:23.963160+0000","last_undegraded":"2021-03-14T04:57:23.963160+0000","last_fullsized":"2021-03-14T04:57:23.963160+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_clean_scrub_stamp":"2021-03-14T04:54:20.170538+0000","log_size":6800,"ondisk_log_size":6800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7679500993,"num_objects":2557,"num_object_clones":0,"num_object_copies":5114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2557,"num_whiteouts":0,"num_read":3925,"num_read_kb":429442,"num_write":8424,"num_write_kb":5020889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"27","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.486867+0000","last_change":"2021-03-14T04:54:12.418875+0000","last_active":"2021-03-14T04:56:58.486867+0000","last_peered":"2021-03-14T04:56:58.486867+0000","last_clean":"2021-03-14T04:56:58.486867+0000","last_became_active":"2021-03-14T04:54:12.418565+0000","last_became_peered":"2021-03-14T04:54:12.418565+0000","last_unstale":"2021-03-14T04:56:58.486867+0000","last_undegraded":"2021-03-14T04:56:58.486867+0000","last_fullsized":"2021-03-14T04:56:58.486867+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"32'5870","reported_seq":"8152","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:57:22.983929+0000","last_change":"2021-03-14T04:54:23.442763+0000","last_active":"2021-03-14T04:57:22.983929+0000","last_peered":"2021-03-14T04:57:22.983929+0000","last_clean":"2021-03-14T04:57:22.983929+0000","last_became_active":"2021-03-14T04:54:23.442127+0000","last_became_peered":"2021-03-14T04:54:23.442127+0000","last_unstale":"2021-03-14T04:57:22.983929+0000","last_undegraded":"2021-03-14T04:57:22.983929+0000","last_fullsized":"2021-03-14T04:57:22.983929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_clean_scrub_stamp":"2021-03-14T04:54:22.413390+0000","log_size":5870,"ondisk_log_size":5870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":123608883,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":11,"num_read":2007,"num_read_kb":998,"num_write":5269,"num_write_kb":1250136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":3919,"num_flush":944,"num_flush_kb":2785170,"num_evict":856,"num_evict_kb":2524988,"num_promote":1052,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"33'6629","reported_seq":"8659","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:24.752363+0000","last_change":"2021-03-14T04:54:21.534780+0000","last_active":"2021-03-14T04:57:24.752363+0000","last_peered":"2021-03-14T04:57:24.752363+0000","last_clean":"2021-03-14T04:57:24.752363+0000","last_became_active":"2021-03-14T04:54:21.533809+0000","last_became_peered":"2021-03-14T04:54:21.533809+0000","last_unstale":"2021-03-14T04:57:24.752363+0000","last_undegraded":"2021-03-14T04:57:24.752363+0000","last_fullsized":"2021-03-14T04:57:24.752363+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_clean_scrub_stamp":"2021-03-14T04:54:20.170538+0000","log_size":6629,"ondisk_log_size":6629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7316861958,"num_objects":2453,"num_object_clones":0,"num_object_copies":4906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2453,"num_whiteouts":0,"num_read":3690,"num_read_kb":429916,"num_write":8238,"num_write_kb":4731837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"26","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.484680+0000","last_change":"2021-03-14T04:54:12.195854+0000","last_active":"2021-03-14T04:56:58.484680+0000","last_peered":"2021-03-14T04:56:58.484680+0000","last_clean":"2021-03-14T04:56:58.484680+0000","last_became_active":"2021-03-14T04:54:12.195060+0000","last_became_peered":"2021-03-14T04:54:12.195060+0000","last_unstale":"2021-03-14T04:56:58.484680+0000","last_undegraded":"2021-03-14T04:56:58.484680+0000","last_fullsized":"2021-03-14T04:56:58.484680+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"26","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.484919+0000","last_change":"2021-03-14T04:54:12.195721+0000","last_active":"2021-03-14T04:56:58.484919+0000","last_peered":"2021-03-14T04:56:58.484919+0000","last_clean":"2021-03-14T04:56:58.484919+0000","last_became_active":"2021-03-14T04:54:12.194809+0000","last_became_peered":"2021-03-14T04:54:12.194809+0000","last_unstale":"2021-03-14T04:56:58.484919+0000","last_undegraded":"2021-03-14T04:56:58.484919+0000","last_fullsized":"2021-03-14T04:56:58.484919+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"27","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.486886+0000","last_change":"2021-03-14T04:54:12.643927+0000","last_active":"2021-03-14T04:56:58.486886+0000","last_peered":"2021-03-14T04:56:58.486886+0000","last_clean":"2021-03-14T04:56:58.486886+0000","last_became_active":"2021-03-14T04:54:12.643112+0000","last_became_peered":"2021-03-14T04:54:12.643112+0000","last_unstale":"2021-03-14T04:56:58.486886+0000","last_undegraded":"2021-03-14T04:56:58.486886+0000","last_fullsized":"2021-03-14T04:56:58.486886+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":485823650,"num_objects":232,"num_object_clones":0,"num_object_copies":464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":204,"num_whiteouts":40,"num_read":7823,"num_read_kb":3822,"num_write":20213,"num_write_kb":4804081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":13882,"num_flush":3617,"num_flush_kb":10535558,"num_evict":3241,"num_evict_kb":9431050,"num_promote":4012,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22590,"ondisk_log_size":22590,"up":8,"acting":8,"num_store_stats":0},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":2,"num_pg":4,"stat_sum":{"num_bytes":29735319058,"num_objects":9940,"num_object_clones":0,"num_object_copies":19880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9940,"num_whiteouts":0,"num_read":15022,"num_read_kb":1498073,"num_write":33027,"num_write_kb":19305998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26626,"ondisk_log_size":26626,"up":8,"acting":8,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640298,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":9873360,"kb_used_data":83864652,"kb_used_omap":12353,"kb_used_meta":0,"kb_avail":83826020,"statfs":{"total":95948165120,"available":85837844480,"internally_reserved":39559168,"allocated":85877403648,"data_stored":85877403648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12649818,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":3,"commit_latency_ns":2000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.505,"15min":0.51500000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.82499999999999996,"5min":1.171,"15min":1.171},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.437,"5min":0.49399999999999999,"15min":0.50800000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.89900000000000002}]},{"osd":1,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.54300000000000004,"15min":0.56799999999999995},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.88600000000000001,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.52300000000000002,"15min":0.54100000000000004},"min":{"1min":0.28599999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.85399999999999998,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.65300000000000002}]},{"osd":2,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.58399999999999996,"15min":0.55500000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":2.681,"5min":2.681,"15min":2.681},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.622,"15min":0.60999999999999999},"min":{"1min":0.32000000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":4.1689999999999996,"5min":4.1689999999999996,"15min":4.1689999999999996},"last":0.98699999999999999}]},{"osd":3,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.502,"15min":0.51700000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.46800000000000003,"15min":0.48599999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.80700000000000005}]},{"osd":4,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.52500000000000002,"15min":0.54500000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.89400000000000002,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.872},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.51700000000000002,"15min":0.53300000000000003},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":1.845,"5min":1.845,"15min":1.845},"last":0.83499999999999996}]}]},{"osd":4,"up_from":11,"seq":47244640299,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":488804,"kb_used_data":93226564,"kb_used_omap":12172,"kb_used_meta":0,"kb_avail":93210576,"statfs":{"total":95948165120,"available":95447629824,"internally_reserved":16371712,"allocated":95464001536,"data_stored":95464001536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12464664,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":2,"commit_latency_ns":1000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.505,"15min":0.54900000000000004},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.69399999999999995,"5min":1.036,"15min":1.036},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.53000000000000003,"15min":0.55400000000000005},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.72399999999999998}]},{"osd":1,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.9790000000000001,"5min":1.9790000000000001,"15min":1.9790000000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.53600000000000003,"15min":0.57099999999999995},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.72699999999999998,"5min":1.075,"15min":1.075},"last":0.75600000000000001}]},{"osd":2,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.52500000000000002,"15min":0.56499999999999995},"min":{"1min":0.314,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.72699999999999998,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.58299999999999996,"15min":0.629},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.91100000000000003,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.56399999999999995}]},{"osd":3,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.46400000000000002,"15min":0.50900000000000001},"min":{"1min":0.23699999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.52600000000000002,"5min":1.202,"15min":1.202},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.48799999999999999,"15min":0.52100000000000002},"min":{"1min":0.246,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.55900000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.629}]},{"osd":5,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.45100000000000001,"15min":0.47299999999999998},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.81299999999999994,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.5,"15min":0.54400000000000004},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.59699999999999998,"5min":1.121,"15min":1.121},"last":0.76500000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640299,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":4952008,"kb_used_data":88768532,"kb_used_omap":3548,"kb_used_meta":0,"kb_avail":88747372,"statfs":{"total":95948165120,"available":90877308928,"internally_reserved":21667840,"allocated":90898976768,"data_stored":90898976768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3633814,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.54700000000000004,"15min":0.59899999999999998},"min":{"1min":0.253,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.89700000000000002,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":1},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.505,"15min":0.53200000000000003},"min":{"1min":0.35299999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.91600000000000004,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":1.2270000000000001}]},{"osd":1,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.52200000000000002,"15min":0.55400000000000005},"min":{"1min":0.27400000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.77800000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.505,"5min":0.53500000000000003,"15min":0.56499999999999995},"min":{"1min":0.29299999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.93200000000000005}]},{"osd":2,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.54000000000000004,"15min":0.57799999999999996},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.68500000000000005,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.55000000000000004,"15min":0.57199999999999995},"min":{"1min":0.34999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.56399999999999995}]},{"osd":4,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.46600000000000003,"15min":0.51100000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.53300000000000003,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":1.1859999999999999},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.47099999999999997,"15min":0.52300000000000002},"min":{"1min":0.247,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.57499999999999996,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.65600000000000003}]},{"osd":5,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.46100000000000002,"15min":0.49199999999999999},"min":{"1min":0.24299999999999999,"5min":0.215,"15min":0.215},"max":{"1min":0.65600000000000003,"5min":0.874,"15min":0.874},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.53200000000000003,"15min":0.57999999999999996},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.69599999999999995,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.751}]}]},{"osd":2,"up_from":11,"seq":47244640298,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":10090296,"kb_used_data":83630492,"kb_used_omap":12460,"kb_used_meta":0,"kb_avail":83609084,"statfs":{"total":95948165120,"available":85615702016,"internally_reserved":21921792,"allocated":85637623808,"data_stored":85637623808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12759532,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":4,"commit_latency_ns":3000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.52400000000000002,"15min":0.48399999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":10.157999999999999,"5min":10.157999999999999,"15min":10.157999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.51600000000000001,"15min":0.502},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.984,"5min":5.984,"15min":5.984},"last":0.51200000000000001}]},{"osd":1,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.67200000000000004,"15min":0.627},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.2140000000000004,"5min":6.2140000000000004,"15min":6.2140000000000004},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.59299999999999997,"15min":0.56599999999999995},"min":{"1min":0.25600000000000001,"5min":0.219,"15min":0.219},"max":{"1min":3.7749999999999999,"5min":3.7749999999999999,"15min":3.7749999999999999},"last":0.49199999999999999}]},{"osd":3,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.70199999999999996,"15min":0.69499999999999995},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":6.2309999999999999,"5min":6.2309999999999999,"15min":6.2309999999999999},"last":0.434},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.71699999999999997,"15min":0.71899999999999997},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":6.0199999999999996,"5min":6.0199999999999996,"15min":6.0199999999999996},"last":0.45500000000000002}]},{"osd":4,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":6.3499999999999996,"5min":6.3499999999999996,"15min":6.3499999999999996},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.68899999999999995,"15min":0.66900000000000004},"min":{"1min":0.379,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":6.3040000000000003,"5min":6.3040000000000003,"15min":6.3040000000000003},"last":0.52200000000000002}]},{"osd":5,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.72099999999999997,"15min":0.72299999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":6.0490000000000004,"5min":6.0490000000000004,"15min":6.0490000000000004},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.69599999999999995,"15min":0.72999999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":0.70099999999999996}]}]},{"osd":0,"up_from":11,"seq":47244640298,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":498748,"kb_used_data":93219896,"kb_used_omap":12182,"kb_used_meta":0,"kb_avail":93200632,"statfs":{"total":95948165120,"available":95437447168,"internally_reserved":19726336,"allocated":95457173504,"data_stored":95457173504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12474636,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":3,"commit_latency_ns":1000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.44900000000000001,"15min":0.45900000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.503},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.53400000000000003,"15min":0.54600000000000004},"min":{"1min":0.251,"5min":0.249,"15min":0.249},"max":{"1min":2.2509999999999999,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":0.61399999999999999}]},{"osd":2,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56499999999999995,"15min":0.59299999999999997},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.351,"5min":2.351,"15min":2.351},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.57399999999999995,"15min":0.63400000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.80900000000000005,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.67400000000000004}]},{"osd":3,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.58899999999999997,"15min":0.60699999999999998},"min":{"1min":0.31900000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":2.4769999999999999,"5min":2.4769999999999999,"15min":2.4769999999999999},"last":0.746},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.64400000000000002,"15min":0.68999999999999995},"min":{"1min":0.36499999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71399999999999997,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.71099999999999997}]},{"osd":4,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.65400000000000003,"15min":0.71699999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71999999999999997,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.70299999999999996,"15min":0.73599999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":2.4409999999999998,"5min":2.4409999999999998,"15min":2.4409999999999998},"last":0.78700000000000003}]},{"osd":5,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.68799999999999994,"15min":0.74299999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.78400000000000003,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.73199999999999998,"15min":0.80800000000000005},"min":{"1min":0.38900000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.77700000000000002,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":0.755}]}]},{"osd":1,"up_from":11,"seq":47244640298,"num_pgs":7,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":14837780,"kb_used_data":78875692,"kb_used_omap":21405,"kb_used_meta":0,"kb_avail":78861600,"statfs":{"total":95948165120,"available":80754278400,"internally_reserved":14430208,"allocated":80768708608,"data_stored":80768708608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21918935,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":3,"commit_latency_ns":2000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.47099999999999997,"15min":0.47999999999999998},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":1.0609999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.308},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.56100000000000005,"15min":0.61299999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.806,"5min":1.806,"15min":1.806},"last":0.34300000000000003}]},{"osd":2,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.501,"15min":0.52800000000000002},"min":{"1min":0.24299999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.60099999999999998,"15min":0.61799999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":2.2570000000000001,"5min":2.2570000000000001,"15min":2.2570000000000001},"last":0.35899999999999999}]},{"osd":3,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.63,"15min":0.69499999999999995},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.0740000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.499},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.621,"15min":0.65500000000000003},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.069,"5min":1.127,"15min":1.127},"last":0.60199999999999998}]},{"osd":4,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.68000000000000005,"15min":0.70399999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.68200000000000005,"15min":0.73799999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.073,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.745}]},{"osd":5,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.68899999999999995,"15min":0.74099999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.73899999999999999,"15min":0.73499999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.1669999999999998,"5min":4.1669999999999998,"15min":4.1669999999999998},"last":0.61499999999999999}]}]}]}}

2021-03-14T15:57:02.226 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T04:58:51.516 INFO:teuthology.orchestra.run:waiting for 300
, 2021-03-14T04:58:51.575 INFO:tasks.ceph.mon.a:Stopped
, 2021-03-14T04:58:51.575 INFO:tasks.ceph:Checking cluster log for badness...

2021-03-14T15:57:02.226 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T04:59:10.571 INFO:teuthology.task.internal.syslog:Checking logs for errors...
, 2021-03-14T04:59:10.572 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com
, 2021-03-14T04:59:10.572 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1

2021-03-14T15:57:02.227 WARNING:scrape:Ignoring malparsed backtrace:     dump_program = dump_out.split("from '")[1].split(' ')[0]
, IndexError: list index out of range
, 2021-03-14T04:59:35.497 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload

2021-03-14T15:57:02.343 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 15.2.9-141-g381387f1 (381387f151822ee4d4927f3fc584ff1af9deb9ce) octopus (stable)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x154) [0x7f4b60d491a0]
,  2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f4b60d4937b]

2021-03-14T15:57:02.345 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T04:57:29.681 INFO:teuthology.orchestra.run.smithi031.stdout:
, dumped all
, 2021-03-14T04:57:29.695 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2021-03-14T04:57:29.378625+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30221142708,"num_objects":10172,"num_object_clones":0,"num_object_copies":20344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10144,"num_whiteouts":40,"num_read":22845,"num_read_kb":1501895,"num_write":53240,"num_write_kb":24110079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":13882,"num_flush":3617,"num_flush_kb":10535558,"num_evict":3241,"num_evict_kb":9431050,"num_promote":4012,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49216,"ondisk_log_size":49216,"up":32,"acting":32,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":32,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":40740996,"kb_used_data":521585828,"kb_used_omap":74122,"kb_used_meta":0,"kb_avail":521455284,"statfs":{"total":575688990720,"available":533970210816,"internally_reserved":133677056,"allocated":534103887872,"data_stored":534103887872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":75901399,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":17,"commit_latency_ns":9000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":279044344,"num_objects":118,"num_object_clones":0,"num_object_copies":236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":21,"num_read":1073,"num_read_kb":969215,"num_write":653,"num_write_kb":525513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":995,"num_flush":84,"num_flush_kb":218105,"num_evict":55,"num_evict_kb":133409,"num_promote":133,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002931"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"27","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.487498+0000","last_change":"2021-03-14T04:54:12.642936+0000","last_active":"2021-03-14T04:56:58.487498+0000","last_peered":"2021-03-14T04:56:58.487498+0000","last_clean":"2021-03-14T04:56:58.487498+0000","last_became_active":"2021-03-14T04:54:12.642317+0000","last_became_peered":"2021-03-14T04:54:12.642317+0000","last_unstale":"2021-03-14T04:56:58.487498+0000","last_undegraded":"2021-03-14T04:56:58.487498+0000","last_fullsized":"2021-03-14T04:56:58.487498+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"26","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.487087+0000","last_change":"2021-03-14T04:54:12.418031+0000","last_active":"2021-03-14T04:56:58.487087+0000","last_peered":"2021-03-14T04:56:58.487087+0000","last_clean":"2021-03-14T04:56:58.487087+0000","last_became_active":"2021-03-14T04:54:12.417412+0000","last_became_peered":"2021-03-14T04:54:12.417412+0000","last_unstale":"2021-03-14T04:56:58.487087+0000","last_undegraded":"2021-03-14T04:56:58.487087+0000","last_fullsized":"2021-03-14T04:56:58.487087+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"32'5843","reported_seq":"8078","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:57:23.366193+0000","last_change":"2021-03-14T04:54:23.443564+0000","last_active":"2021-03-14T04:57:23.366193+0000","last_peered":"2021-03-14T04:57:23.366193+0000","last_clean":"2021-03-14T04:57:23.366193+0000","last_became_active":"2021-03-14T04:54:23.443237+0000","last_became_peered":"2021-03-14T04:54:23.443237+0000","last_unstale":"2021-03-14T04:57:23.366193+0000","last_undegraded":"2021-03-14T04:57:23.366193+0000","last_fullsized":"2021-03-14T04:57:23.366193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_clean_scrub_stamp":"2021-03-14T04:54:22.413390+0000","log_size":5843,"ondisk_log_size":5843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":138267928,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":11,"num_read":1974,"num_read_kb":990,"num_write":5143,"num_write_kb":1198886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":3937,"num_flush":923,"num_flush_kb":2682369,"num_evict":856,"num_evict_kb":2480754,"num_promote":1047,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"33'6710","reported_seq":"8801","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:24.300664+0000","last_change":"2021-03-14T04:54:21.535219+0000","last_active":"2021-03-14T04:57:24.300664+0000","last_peered":"2021-03-14T04:57:24.300664+0000","last_clean":"2021-03-14T04:57:24.300664+0000","last_became_active":"2021-03-14T04:54:21.534448+0000","last_became_peered":"2021-03-14T04:54:21.534448+0000","last_unstale":"2021-03-14T04:57:24.300664+0000","last_undegraded":"2021-03-14T04:57:24.300664+0000","last_fullsized":"2021-03-14T04:57:24.300664+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_clean_scrub_stamp":"2021-03-14T04:54:20.170538+0000","log_size":6710,"ondisk_log_size":6710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7433652118,"num_objects":2487,"num_object_clones":0,"num_object_copies":4974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2487,"num_whiteouts":0,"num_read":3750,"num_read_kb":341397,"num_write":8348,"num_write_kb":4729158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"27","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.486920+0000","last_change":"2021-03-14T04:54:12.419664+0000","last_active":"2021-03-14T04:56:58.486920+0000","last_peered":"2021-03-14T04:56:58.486920+0000","last_clean":"2021-03-14T04:56:58.486920+0000","last_became_active":"2021-03-14T04:54:12.418921+0000","last_became_peered":"2021-03-14T04:54:12.418921+0000","last_unstale":"2021-03-14T04:56:58.486920+0000","last_undegraded":"2021-03-14T04:56:58.486920+0000","last_fullsized":"2021-03-14T04:56:58.486920+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"33'5405","reported_seq":"7509","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:24.302411+0000","last_change":"2021-03-14T04:54:23.478628+0000","last_active":"2021-03-14T04:57:24.302411+0000","last_peered":"2021-03-14T04:57:24.302411+0000","last_clean":"2021-03-14T04:57:24.302411+0000","last_became_active":"2021-03-14T04:54:23.477906+0000","last_became_peered":"2021-03-14T04:54:23.477906+0000","last_unstale":"2021-03-14T04:57:24.302411+0000","last_undegraded":"2021-03-14T04:57:24.302411+0000","last_fullsized":"2021-03-14T04:57:24.302411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_clean_scrub_stamp":"2021-03-14T04:54:22.413390+0000","log_size":5405,"ondisk_log_size":5405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":109356376,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":9,"num_read":1885,"num_read_kb":916,"num_write":4890,"num_write_kb":1177491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":3022,"num_flush":870,"num_flush_kb":2517847,"num_evict":799,"num_evict_kb":2306754,"num_promote":955,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"33'6487","reported_seq":"8657","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:23.833806+0000","last_change":"2021-03-14T04:54:21.421319+0000","last_active":"2021-03-14T04:57:23.833806+0000","last_peered":"2021-03-14T04:57:23.833806+0000","last_clean":"2021-03-14T04:57:23.833806+0000","last_became_active":"2021-03-14T04:54:21.420795+0000","last_became_peered":"2021-03-14T04:54:21.420795+0000","last_unstale":"2021-03-14T04:57:23.833806+0000","last_undegraded":"2021-03-14T04:57:23.833806+0000","last_fullsized":"2021-03-14T04:57:23.833806+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_clean_scrub_stamp":"2021-03-14T04:54:20.170538+0000","log_size":6487,"ondisk_log_size":6487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7305303989,"num_objects":2443,"num_object_clones":0,"num_object_copies":4886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2443,"num_whiteouts":0,"num_read":3657,"num_read_kb":297318,"num_write":8017,"num_write_kb":4824114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"26","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.487176+0000","last_change":"2021-03-14T04:54:12.418478+0000","last_active":"2021-03-14T04:56:58.487176+0000","last_peered":"2021-03-14T04:56:58.487176+0000","last_clean":"2021-03-14T04:56:58.487176+0000","last_became_active":"2021-03-14T04:54:12.417271+0000","last_became_peered":"2021-03-14T04:54:12.417271+0000","last_unstale":"2021-03-14T04:56:58.487176+0000","last_undegraded":"2021-03-14T04:56:58.487176+0000","last_fullsized":"2021-03-14T04:56:58.487176+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"33'5472","reported_seq":"7685","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:24.757059+0000","last_change":"2021-03-14T04:54:23.645513+0000","last_active":"2021-03-14T04:57:24.757059+0000","last_peered":"2021-03-14T04:57:24.757059+0000","last_clean":"2021-03-14T04:57:24.757059+0000","last_became_active":"2021-03-14T04:54:23.645209+0000","last_became_peered":"2021-03-14T04:54:23.645209+0000","last_unstale":"2021-03-14T04:57:24.757059+0000","last_undegraded":"2021-03-14T04:57:24.757059+0000","last_fullsized":"2021-03-14T04:57:24.757059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_clean_scrub_stamp":"2021-03-14T04:54:22.413390+0000","log_size":5472,"ondisk_log_size":5472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":114590463,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":9,"num_read":1957,"num_read_kb":918,"num_write":4911,"num_write_kb":1177568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":3004,"num_flush":880,"num_flush_kb":2550172,"num_evict":730,"num_evict_kb":2118554,"num_promote":958,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"33'6800","reported_seq":"9068","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:23.963160+0000","last_change":"2021-03-14T04:54:21.532921+0000","last_active":"2021-03-14T04:57:23.963160+0000","last_peered":"2021-03-14T04:57:23.963160+0000","last_clean":"2021-03-14T04:57:23.963160+0000","last_became_active":"2021-03-14T04:54:21.532542+0000","last_became_peered":"2021-03-14T04:54:21.532542+0000","last_unstale":"2021-03-14T04:57:23.963160+0000","last_undegraded":"2021-03-14T04:57:23.963160+0000","last_fullsized":"2021-03-14T04:57:23.963160+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_clean_scrub_stamp":"2021-03-14T04:54:20.170538+0000","log_size":6800,"ondisk_log_size":6800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7679500993,"num_objects":2557,"num_object_clones":0,"num_object_copies":5114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2557,"num_whiteouts":0,"num_read":3925,"num_read_kb":429442,"num_write":8424,"num_write_kb":5020889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"27","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.486867+0000","last_change":"2021-03-14T04:54:12.418875+0000","last_active":"2021-03-14T04:56:58.486867+0000","last_peered":"2021-03-14T04:56:58.486867+0000","last_clean":"2021-03-14T04:56:58.486867+0000","last_became_active":"2021-03-14T04:54:12.418565+0000","last_became_peered":"2021-03-14T04:54:12.418565+0000","last_unstale":"2021-03-14T04:56:58.486867+0000","last_undegraded":"2021-03-14T04:56:58.486867+0000","last_fullsized":"2021-03-14T04:56:58.486867+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"32'5870","reported_seq":"8152","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:57:22.983929+0000","last_change":"2021-03-14T04:54:23.442763+0000","last_active":"2021-03-14T04:57:22.983929+0000","last_peered":"2021-03-14T04:57:22.983929+0000","last_clean":"2021-03-14T04:57:22.983929+0000","last_became_active":"2021-03-14T04:54:23.442127+0000","last_became_peered":"2021-03-14T04:54:23.442127+0000","last_unstale":"2021-03-14T04:57:22.983929+0000","last_undegraded":"2021-03-14T04:57:22.983929+0000","last_fullsized":"2021-03-14T04:57:22.983929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_clean_scrub_stamp":"2021-03-14T04:54:22.413390+0000","log_size":5870,"ondisk_log_size":5870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":123608883,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":11,"num_read":2007,"num_read_kb":998,"num_write":5269,"num_write_kb":1250136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":3919,"num_flush":944,"num_flush_kb":2785170,"num_evict":856,"num_evict_kb":2524988,"num_promote":1052,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"33'6629","reported_seq":"8659","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:24.752363+0000","last_change":"2021-03-14T04:54:21.534780+0000","last_active":"2021-03-14T04:57:24.752363+0000","last_peered":"2021-03-14T04:57:24.752363+0000","last_clean":"2021-03-14T04:57:24.752363+0000","last_became_active":"2021-03-14T04:54:21.533809+0000","last_became_peered":"2021-03-14T04:54:21.533809+0000","last_unstale":"2021-03-14T04:57:24.752363+0000","last_undegraded":"2021-03-14T04:57:24.752363+0000","last_fullsized":"2021-03-14T04:57:24.752363+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_clean_scrub_stamp":"2021-03-14T04:54:20.170538+0000","log_size":6629,"ondisk_log_size":6629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7316861958,"num_objects":2453,"num_object_clones":0,"num_object_copies":4906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2453,"num_whiteouts":0,"num_read":3690,"num_read_kb":429916,"num_write":8238,"num_write_kb":4731837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"26","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.484680+0000","last_change":"2021-03-14T04:54:12.195854+0000","last_active":"2021-03-14T04:56:58.484680+0000","last_peered":"2021-03-14T04:56:58.484680+0000","last_clean":"2021-03-14T04:56:58.484680+0000","last_became_active":"2021-03-14T04:54:12.195060+0000","last_became_peered":"2021-03-14T04:54:12.195060+0000","last_unstale":"2021-03-14T04:56:58.484680+0000","last_undegraded":"2021-03-14T04:56:58.484680+0000","last_fullsized":"2021-03-14T04:56:58.484680+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"26","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.484919+0000","last_change":"2021-03-14T04:54:12.195721+0000","last_active":"2021-03-14T04:56:58.484919+0000","last_peered":"2021-03-14T04:56:58.484919+0000","last_clean":"2021-03-14T04:56:58.484919+0000","last_became_active":"2021-03-14T04:54:12.194809+0000","last_became_peered":"2021-03-14T04:54:12.194809+0000","last_unstale":"2021-03-14T04:56:58.484919+0000","last_undegraded":"2021-03-14T04:56:58.484919+0000","last_fullsized":"2021-03-14T04:56:58.484919+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"27","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.486886+0000","last_change":"2021-03-14T04:54:12.643927+0000","last_active":"2021-03-14T04:56:58.486886+0000","last_peered":"2021-03-14T04:56:58.486886+0000","last_clean":"2021-03-14T04:56:58.486886+0000","last_became_active":"2021-03-14T04:54:12.643112+0000","last_became_peered":"2021-03-14T04:54:12.643112+0000","last_unstale":"2021-03-14T04:56:58.486886+0000","last_undegraded":"2021-03-14T04:56:58.486886+0000","last_fullsized":"2021-03-14T04:56:58.486886+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":485823650,"num_objects":232,"num_object_clones":0,"num_object_copies":464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":204,"num_whiteouts":40,"num_read":7823,"num_read_kb":3822,"num_write":20213,"num_write_kb":4804081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":13882,"num_flush":3617,"num_flush_kb":10535558,"num_evict":3241,"num_evict_kb":9431050,"num_promote":4012,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22590,"ondisk_log_size":22590,"up":8,"acting":8,"num_store_stats":0},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":2,"num_pg":4,"stat_sum":{"num_bytes":29735319058,"num_objects":9940,"num_object_clones":0,"num_object_copies":19880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9940,"num_whiteouts":0,"num_read":15022,"num_read_kb":1498073,"num_write":33027,"num_write_kb":19305998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26626,"ondisk_log_size":26626,"up":8,"acting":8,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640298,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":9873360,"kb_used_data":83864652,"kb_used_omap":12353,"kb_used_meta":0,"kb_avail":83826020,"statfs":{"total":95948165120,"available":85837844480,"internally_reserved":39559168,"allocated":85877403648,"data_stored":85877403648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12649818,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":3,"commit_latency_ns":2000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.505,"15min":0.51500000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.82499999999999996,"5min":1.171,"15min":1.171},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.437,"5min":0.49399999999999999,"15min":0.50800000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.89900000000000002}]},{"osd":1,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.54300000000000004,"15min":0.56799999999999995},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.88600000000000001,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.52300000000000002,"15min":0.54100000000000004},"min":{"1min":0.28599999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.85399999999999998,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.65300000000000002}]},{"osd":2,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.58399999999999996,"15min":0.55500000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":2.681,"5min":2.681,"15min":2.681},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.622,"15min":0.60999999999999999},"min":{"1min":0.32000000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":4.1689999999999996,"5min":4.1689999999999996,"15min":4.1689999999999996},"last":0.98699999999999999}]},{"osd":3,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.502,"15min":0.51700000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.46800000000000003,"15min":0.48599999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.80700000000000005}]},{"osd":4,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.52500000000000002,"15min":0.54500000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.89400000000000002,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.872},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.51700000000000002,"15min":0.53300000000000003},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":1.845,"5min":1.845,"15min":1.845},"last":0.83499999999999996}]}]},{"osd":4,"up_from":11,"seq":47244640299,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":488804,"kb_used_data":93226564,"kb_used_omap":12172,"kb_used_meta":0,"kb_avail":93210576,"statfs":{"total":95948165120,"available":95447629824,"internally_reserved":16371712,"allocated":95464001536,"data_stored":95464001536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12464664,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":2,"commit_latency_ns":1000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.505,"15min":0.54900000000000004},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.69399999999999995,"5min":1.036,"15min":1.036},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.53000000000000003,"15min":0.55400000000000005},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.72399999999999998}]},{"osd":1,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.9790000000000001,"5min":1.9790000000000001,"15min":1.9790000000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.53600000000000003,"15min":0.57099999999999995},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.72699999999999998,"5min":1.075,"15min":1.075},"last":0.75600000000000001}]},{"osd":2,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.52500000000000002,"15min":0.56499999999999995},"min":{"1min":0.314,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.72699999999999998,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.58299999999999996,"15min":0.629},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.91100000000000003,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.56399999999999995}]},{"osd":3,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.46400000000000002,"15min":0.50900000000000001},"min":{"1min":0.23699999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.52600000000000002,"5min":1.202,"15min":1.202},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.48799999999999999,"15min":0.52100000000000002},"min":{"1min":0.246,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.55900000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.629}]},{"osd":5,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.45100000000000001,"15min":0.47299999999999998},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.81299999999999994,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.5,"15min":0.54400000000000004},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.59699999999999998,"5min":1.121,"15min":1.121},"last":0.76500000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640299,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":4952008,"kb_used_data":88768532,"kb_used_omap":3548,"kb_used_meta":0,"kb_avail":88747372,"statfs":{"total":95948165120,"available":90877308928,"internally_reserved":21667840,"allocated":90898976768,"data_stored":90898976768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3633814,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.54700000000000004,"15min":0.59899999999999998},"min":{"1min":0.253,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.89700000000000002,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":1},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.505,"15min":0.53200000000000003},"min":{"1min":0.35299999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.91600000000000004,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":1.2270000000000001}]},{"osd":1,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.52200000000000002,"15min":0.55400000000000005},"min":{"1min":0.27400000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.77800000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.505,"5min":0.53500000000000003,"15min":0.56499999999999995},"min":{"1min":0.29299999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.93200000000000005}]},{"osd":2,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.54000000000000004,"15min":0.57799999999999996},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.68500000000000005,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.55000000000000004,"15min":0.57199999999999995},"min":{"1min":0.34999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.56399999999999995}]},{"osd":4,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.46600000000000003,"15min":0.51100000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.53300000000000003,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":1.1859999999999999},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.47099999999999997,"15min":0.52300000000000002},"min":{"1min":0.247,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.57499999999999996,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.65600000000000003}]},{"osd":5,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.46100000000000002,"15min":0.49199999999999999},"min":{"1min":0.24299999999999999,"5min":0.215,"15min":0.215},"max":{"1min":0.65600000000000003,"5min":0.874,"15min":0.874},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.53200000000000003,"15min":0.57999999999999996},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.69599999999999995,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.751}]}]},{"osd":2,"up_from":11,"seq":47244640298,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":10090296,"kb_used_data":83630492,"kb_used_omap":12460,"kb_used_meta":0,"kb_avail":83609084,"statfs":{"total":95948165120,"available":85615702016,"internally_reserved":21921792,"allocated":85637623808,"data_stored":85637623808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12759532,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":4,"commit_latency_ns":3000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.52400000000000002,"15min":0.48399999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":10.157999999999999,"5min":10.157999999999999,"15min":10.157999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.51600000000000001,"15min":0.502},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.984,"5min":5.984,"15min":5.984},"last":0.51200000000000001}]},{"osd":1,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.67200000000000004,"15min":0.627},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.2140000000000004,"5min":6.2140000000000004,"15min":6.2140000000000004},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.59299999999999997,"15min":0.56599999999999995},"min":{"1min":0.25600000000000001,"5min":0.219,"15min":0.219},"max":{"1min":3.7749999999999999,"5min":3.7749999999999999,"15min":3.7749999999999999},"last":0.49199999999999999}]},{"osd":3,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.70199999999999996,"15min":0.69499999999999995},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":6.2309999999999999,"5min":6.2309999999999999,"15min":6.2309999999999999},"last":0.434},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.71699999999999997,"15min":0.71899999999999997},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":6.0199999999999996,"5min":6.0199999999999996,"15min":6.0199999999999996},"last":0.45500000000000002}]},{"osd":4,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":6.3499999999999996,"5min":6.3499999999999996,"15min":6.3499999999999996},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.68899999999999995,"15min":0.66900000000000004},"min":{"1min":0.379,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":6.3040000000000003,"5min":6.3040000000000003,"15min":6.3040000000000003},"last":0.52200000000000002}]},{"osd":5,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.72099999999999997,"15min":0.72299999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":6.0490000000000004,"5min":6.0490000000000004,"15min":6.0490000000000004},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.69599999999999995,"15min":0.72999999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":0.70099999999999996}]}]},{"osd":0,"up_from":11,"seq":47244640298,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":498748,"kb_used_data":93219896,"kb_used_omap":12182,"kb_used_meta":0,"kb_avail":93200632,"statfs":{"total":95948165120,"available":95437447168,"internally_reserved":19726336,"allocated":95457173504,"data_stored":95457173504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12474636,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":3,"commit_latency_ns":1000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.44900000000000001,"15min":0.45900000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.503},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.53400000000000003,"15min":0.54600000000000004},"min":{"1min":0.251,"5min":0.249,"15min":0.249},"max":{"1min":2.2509999999999999,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":0.61399999999999999}]},{"osd":2,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56499999999999995,"15min":0.59299999999999997},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.351,"5min":2.351,"15min":2.351},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.57399999999999995,"15min":0.63400000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.80900000000000005,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.67400000000000004}]},{"osd":3,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.58899999999999997,"15min":0.60699999999999998},"min":{"1min":0.31900000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":2.4769999999999999,"5min":2.4769999999999999,"15min":2.4769999999999999},"last":0.746},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.64400000000000002,"15min":0.68999999999999995},"min":{"1min":0.36499999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71399999999999997,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.71099999999999997}]},{"osd":4,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.65400000000000003,"15min":0.71699999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71999999999999997,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.70299999999999996,"15min":0.73599999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":2.4409999999999998,"5min":2.4409999999999998,"15min":2.4409999999999998},"last":0.78700000000000003}]},{"osd":5,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.68799999999999994,"15min":0.74299999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.78400000000000003,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.73199999999999998,"15min":0.80800000000000005},"min":{"1min":0.38900000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.77700000000000002,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":0.755}]}]},{"osd":1,"up_from":11,"seq":47244640298,"num_pgs":7,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":14837780,"kb_used_data":78875692,"kb_used_omap":21405,"kb_used_meta":0,"kb_avail":78861600,"statfs":{"total":95948165120,"available":80754278400,"internally_reserved":14430208,"allocated":80768708608,"data_stored":80768708608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21918935,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":3,"commit_latency_ns":2000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.47099999999999997,"15min":0.47999999999999998},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":1.0609999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.308},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.56100000000000005,"15min":0.61299999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.806,"5min":1.806,"15min":1.806},"last":0.34300000000000003}]},{"osd":2,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.501,"15min":0.52800000000000002},"min":{"1min":0.24299999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.60099999999999998,"15min":0.61799999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":2.2570000000000001,"5min":2.2570000000000001,"15min":2.2570000000000001},"last":0.35899999999999999}]},{"osd":3,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.63,"15min":0.69499999999999995},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.0740000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.499},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.621,"15min":0.65500000000000003},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.069,"5min":1.127,"15min":1.127},"last":0.60199999999999998}]},{"osd":4,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.68000000000000005,"15min":0.70399999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.68200000000000005,"15min":0.73799999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.073,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.745}]},{"osd":5,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.68899999999999995,"15min":0.74099999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.73899999999999999,"15min":0.73499999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.1669999999999998,"5min":4.1669999999999998,"15min":4.1669999999999998},"last":0.61499999999999999}]}]}]}}

2021-03-14T15:57:02.347 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T04:58:51.516 INFO:teuthology.orchestra.run:waiting for 300
, 2021-03-14T04:58:51.575 INFO:tasks.ceph.mon.a:Stopped
, 2021-03-14T04:58:51.575 INFO:tasks.ceph:Checking cluster log for badness...

2021-03-14T15:57:02.347 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T04:59:10.571 INFO:teuthology.task.internal.syslog:Checking logs for errors...
, 2021-03-14T04:59:10.572 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com
, 2021-03-14T04:59:10.572 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1

2021-03-14T15:57:02.348 WARNING:scrape:Ignoring malparsed backtrace:     dump_program = dump_out.split("from '")[1].split(' ')[0]
, IndexError: list index out of range
, 2021-03-14T04:59:35.497 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload

2021-03-14T15:57:02.461 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 15.2.9-141-g381387f1 (381387f151822ee4d4927f3fc584ff1af9deb9ce) octopus (stable)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x154) [0x7f4b60d491a0]
,  2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f4b60d4937b]

2021-03-14T15:57:02.464 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T04:57:29.681 INFO:teuthology.orchestra.run.smithi031.stdout:
, dumped all
, 2021-03-14T04:57:29.695 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2021-03-14T04:57:29.378625+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30221142708,"num_objects":10172,"num_object_clones":0,"num_object_copies":20344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10144,"num_whiteouts":40,"num_read":22845,"num_read_kb":1501895,"num_write":53240,"num_write_kb":24110079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":13882,"num_flush":3617,"num_flush_kb":10535558,"num_evict":3241,"num_evict_kb":9431050,"num_promote":4012,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49216,"ondisk_log_size":49216,"up":32,"acting":32,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":32,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":40740996,"kb_used_data":521585828,"kb_used_omap":74122,"kb_used_meta":0,"kb_avail":521455284,"statfs":{"total":575688990720,"available":533970210816,"internally_reserved":133677056,"allocated":534103887872,"data_stored":534103887872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":75901399,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":17,"commit_latency_ns":9000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":279044344,"num_objects":118,"num_object_clones":0,"num_object_copies":236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":21,"num_read":1073,"num_read_kb":969215,"num_write":653,"num_write_kb":525513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":995,"num_flush":84,"num_flush_kb":218105,"num_evict":55,"num_evict_kb":133409,"num_promote":133,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002931"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"27","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.487498+0000","last_change":"2021-03-14T04:54:12.642936+0000","last_active":"2021-03-14T04:56:58.487498+0000","last_peered":"2021-03-14T04:56:58.487498+0000","last_clean":"2021-03-14T04:56:58.487498+0000","last_became_active":"2021-03-14T04:54:12.642317+0000","last_became_peered":"2021-03-14T04:54:12.642317+0000","last_unstale":"2021-03-14T04:56:58.487498+0000","last_undegraded":"2021-03-14T04:56:58.487498+0000","last_fullsized":"2021-03-14T04:56:58.487498+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"26","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.487087+0000","last_change":"2021-03-14T04:54:12.418031+0000","last_active":"2021-03-14T04:56:58.487087+0000","last_peered":"2021-03-14T04:56:58.487087+0000","last_clean":"2021-03-14T04:56:58.487087+0000","last_became_active":"2021-03-14T04:54:12.417412+0000","last_became_peered":"2021-03-14T04:54:12.417412+0000","last_unstale":"2021-03-14T04:56:58.487087+0000","last_undegraded":"2021-03-14T04:56:58.487087+0000","last_fullsized":"2021-03-14T04:56:58.487087+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"32'5843","reported_seq":"8078","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:57:23.366193+0000","last_change":"2021-03-14T04:54:23.443564+0000","last_active":"2021-03-14T04:57:23.366193+0000","last_peered":"2021-03-14T04:57:23.366193+0000","last_clean":"2021-03-14T04:57:23.366193+0000","last_became_active":"2021-03-14T04:54:23.443237+0000","last_became_peered":"2021-03-14T04:54:23.443237+0000","last_unstale":"2021-03-14T04:57:23.366193+0000","last_undegraded":"2021-03-14T04:57:23.366193+0000","last_fullsized":"2021-03-14T04:57:23.366193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_clean_scrub_stamp":"2021-03-14T04:54:22.413390+0000","log_size":5843,"ondisk_log_size":5843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":138267928,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":11,"num_read":1974,"num_read_kb":990,"num_write":5143,"num_write_kb":1198886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":3937,"num_flush":923,"num_flush_kb":2682369,"num_evict":856,"num_evict_kb":2480754,"num_promote":1047,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"33'6710","reported_seq":"8801","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:24.300664+0000","last_change":"2021-03-14T04:54:21.535219+0000","last_active":"2021-03-14T04:57:24.300664+0000","last_peered":"2021-03-14T04:57:24.300664+0000","last_clean":"2021-03-14T04:57:24.300664+0000","last_became_active":"2021-03-14T04:54:21.534448+0000","last_became_peered":"2021-03-14T04:54:21.534448+0000","last_unstale":"2021-03-14T04:57:24.300664+0000","last_undegraded":"2021-03-14T04:57:24.300664+0000","last_fullsized":"2021-03-14T04:57:24.300664+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_clean_scrub_stamp":"2021-03-14T04:54:20.170538+0000","log_size":6710,"ondisk_log_size":6710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7433652118,"num_objects":2487,"num_object_clones":0,"num_object_copies":4974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2487,"num_whiteouts":0,"num_read":3750,"num_read_kb":341397,"num_write":8348,"num_write_kb":4729158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"27","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.486920+0000","last_change":"2021-03-14T04:54:12.419664+0000","last_active":"2021-03-14T04:56:58.486920+0000","last_peered":"2021-03-14T04:56:58.486920+0000","last_clean":"2021-03-14T04:56:58.486920+0000","last_became_active":"2021-03-14T04:54:12.418921+0000","last_became_peered":"2021-03-14T04:54:12.418921+0000","last_unstale":"2021-03-14T04:56:58.486920+0000","last_undegraded":"2021-03-14T04:56:58.486920+0000","last_fullsized":"2021-03-14T04:56:58.486920+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"33'5405","reported_seq":"7509","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:24.302411+0000","last_change":"2021-03-14T04:54:23.478628+0000","last_active":"2021-03-14T04:57:24.302411+0000","last_peered":"2021-03-14T04:57:24.302411+0000","last_clean":"2021-03-14T04:57:24.302411+0000","last_became_active":"2021-03-14T04:54:23.477906+0000","last_became_peered":"2021-03-14T04:54:23.477906+0000","last_unstale":"2021-03-14T04:57:24.302411+0000","last_undegraded":"2021-03-14T04:57:24.302411+0000","last_fullsized":"2021-03-14T04:57:24.302411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_clean_scrub_stamp":"2021-03-14T04:54:22.413390+0000","log_size":5405,"ondisk_log_size":5405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":109356376,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":9,"num_read":1885,"num_read_kb":916,"num_write":4890,"num_write_kb":1177491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":3022,"num_flush":870,"num_flush_kb":2517847,"num_evict":799,"num_evict_kb":2306754,"num_promote":955,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"33'6487","reported_seq":"8657","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:23.833806+0000","last_change":"2021-03-14T04:54:21.421319+0000","last_active":"2021-03-14T04:57:23.833806+0000","last_peered":"2021-03-14T04:57:23.833806+0000","last_clean":"2021-03-14T04:57:23.833806+0000","last_became_active":"2021-03-14T04:54:21.420795+0000","last_became_peered":"2021-03-14T04:54:21.420795+0000","last_unstale":"2021-03-14T04:57:23.833806+0000","last_undegraded":"2021-03-14T04:57:23.833806+0000","last_fullsized":"2021-03-14T04:57:23.833806+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_clean_scrub_stamp":"2021-03-14T04:54:20.170538+0000","log_size":6487,"ondisk_log_size":6487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7305303989,"num_objects":2443,"num_object_clones":0,"num_object_copies":4886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2443,"num_whiteouts":0,"num_read":3657,"num_read_kb":297318,"num_write":8017,"num_write_kb":4824114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"26","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.487176+0000","last_change":"2021-03-14T04:54:12.418478+0000","last_active":"2021-03-14T04:56:58.487176+0000","last_peered":"2021-03-14T04:56:58.487176+0000","last_clean":"2021-03-14T04:56:58.487176+0000","last_became_active":"2021-03-14T04:54:12.417271+0000","last_became_peered":"2021-03-14T04:54:12.417271+0000","last_unstale":"2021-03-14T04:56:58.487176+0000","last_undegraded":"2021-03-14T04:56:58.487176+0000","last_fullsized":"2021-03-14T04:56:58.487176+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"33'5472","reported_seq":"7685","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:24.757059+0000","last_change":"2021-03-14T04:54:23.645513+0000","last_active":"2021-03-14T04:57:24.757059+0000","last_peered":"2021-03-14T04:57:24.757059+0000","last_clean":"2021-03-14T04:57:24.757059+0000","last_became_active":"2021-03-14T04:54:23.645209+0000","last_became_peered":"2021-03-14T04:54:23.645209+0000","last_unstale":"2021-03-14T04:57:24.757059+0000","last_undegraded":"2021-03-14T04:57:24.757059+0000","last_fullsized":"2021-03-14T04:57:24.757059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_clean_scrub_stamp":"2021-03-14T04:54:22.413390+0000","log_size":5472,"ondisk_log_size":5472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":114590463,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":9,"num_read":1957,"num_read_kb":918,"num_write":4911,"num_write_kb":1177568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":3004,"num_flush":880,"num_flush_kb":2550172,"num_evict":730,"num_evict_kb":2118554,"num_promote":958,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"33'6800","reported_seq":"9068","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:23.963160+0000","last_change":"2021-03-14T04:54:21.532921+0000","last_active":"2021-03-14T04:57:23.963160+0000","last_peered":"2021-03-14T04:57:23.963160+0000","last_clean":"2021-03-14T04:57:23.963160+0000","last_became_active":"2021-03-14T04:54:21.532542+0000","last_became_peered":"2021-03-14T04:54:21.532542+0000","last_unstale":"2021-03-14T04:57:23.963160+0000","last_undegraded":"2021-03-14T04:57:23.963160+0000","last_fullsized":"2021-03-14T04:57:23.963160+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_clean_scrub_stamp":"2021-03-14T04:54:20.170538+0000","log_size":6800,"ondisk_log_size":6800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7679500993,"num_objects":2557,"num_object_clones":0,"num_object_copies":5114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2557,"num_whiteouts":0,"num_read":3925,"num_read_kb":429442,"num_write":8424,"num_write_kb":5020889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"27","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.486867+0000","last_change":"2021-03-14T04:54:12.418875+0000","last_active":"2021-03-14T04:56:58.486867+0000","last_peered":"2021-03-14T04:56:58.486867+0000","last_clean":"2021-03-14T04:56:58.486867+0000","last_became_active":"2021-03-14T04:54:12.418565+0000","last_became_peered":"2021-03-14T04:54:12.418565+0000","last_unstale":"2021-03-14T04:56:58.486867+0000","last_undegraded":"2021-03-14T04:56:58.486867+0000","last_fullsized":"2021-03-14T04:56:58.486867+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"32'5870","reported_seq":"8152","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:57:22.983929+0000","last_change":"2021-03-14T04:54:23.442763+0000","last_active":"2021-03-14T04:57:22.983929+0000","last_peered":"2021-03-14T04:57:22.983929+0000","last_clean":"2021-03-14T04:57:22.983929+0000","last_became_active":"2021-03-14T04:54:23.442127+0000","last_became_peered":"2021-03-14T04:54:23.442127+0000","last_unstale":"2021-03-14T04:57:22.983929+0000","last_undegraded":"2021-03-14T04:57:22.983929+0000","last_fullsized":"2021-03-14T04:57:22.983929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:22.413390+0000","last_clean_scrub_stamp":"2021-03-14T04:54:22.413390+0000","log_size":5870,"ondisk_log_size":5870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":123608883,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":11,"num_read":2007,"num_read_kb":998,"num_write":5269,"num_write_kb":1250136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":8,"num_bytes_hit_set_archive":3919,"num_flush":944,"num_flush_kb":2785170,"num_evict":856,"num_evict_kb":2524988,"num_promote":1052,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"33'6629","reported_seq":"8659","reported_epoch":"33","state":"active+clean","last_fresh":"2021-03-14T04:57:24.752363+0000","last_change":"2021-03-14T04:54:21.534780+0000","last_active":"2021-03-14T04:57:24.752363+0000","last_peered":"2021-03-14T04:57:24.752363+0000","last_clean":"2021-03-14T04:57:24.752363+0000","last_became_active":"2021-03-14T04:54:21.533809+0000","last_became_peered":"2021-03-14T04:54:21.533809+0000","last_unstale":"2021-03-14T04:57:24.752363+0000","last_undegraded":"2021-03-14T04:57:24.752363+0000","last_fullsized":"2021-03-14T04:57:24.752363+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:20.170538+0000","last_clean_scrub_stamp":"2021-03-14T04:54:20.170538+0000","log_size":6629,"ondisk_log_size":6629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7316861958,"num_objects":2453,"num_object_clones":0,"num_object_copies":4906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2453,"num_whiteouts":0,"num_read":3690,"num_read_kb":429916,"num_write":8238,"num_write_kb":4731837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"26","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.484680+0000","last_change":"2021-03-14T04:54:12.195854+0000","last_active":"2021-03-14T04:56:58.484680+0000","last_peered":"2021-03-14T04:56:58.484680+0000","last_clean":"2021-03-14T04:56:58.484680+0000","last_became_active":"2021-03-14T04:54:12.195060+0000","last_became_peered":"2021-03-14T04:54:12.195060+0000","last_unstale":"2021-03-14T04:56:58.484680+0000","last_undegraded":"2021-03-14T04:56:58.484680+0000","last_fullsized":"2021-03-14T04:56:58.484680+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"26","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.484919+0000","last_change":"2021-03-14T04:54:12.195721+0000","last_active":"2021-03-14T04:56:58.484919+0000","last_peered":"2021-03-14T04:56:58.484919+0000","last_clean":"2021-03-14T04:56:58.484919+0000","last_became_active":"2021-03-14T04:54:12.194809+0000","last_became_peered":"2021-03-14T04:54:12.194809+0000","last_unstale":"2021-03-14T04:56:58.484919+0000","last_undegraded":"2021-03-14T04:56:58.484919+0000","last_fullsized":"2021-03-14T04:56:58.484919+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"27","reported_epoch":"32","state":"active+clean","last_fresh":"2021-03-14T04:56:58.486886+0000","last_change":"2021-03-14T04:54:12.643927+0000","last_active":"2021-03-14T04:56:58.486886+0000","last_peered":"2021-03-14T04:56:58.486886+0000","last_clean":"2021-03-14T04:56:58.486886+0000","last_became_active":"2021-03-14T04:54:12.643112+0000","last_became_peered":"2021-03-14T04:54:12.643112+0000","last_unstale":"2021-03-14T04:56:58.486886+0000","last_undegraded":"2021-03-14T04:56:58.486886+0000","last_fullsized":"2021-03-14T04:56:58.486886+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-14T04:54:11.173644+0000","last_clean_scrub_stamp":"2021-03-14T04:54:11.173644+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":485823650,"num_objects":232,"num_object_clones":0,"num_object_copies":464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":204,"num_whiteouts":40,"num_read":7823,"num_read_kb":3822,"num_write":20213,"num_write_kb":4804081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":28,"num_bytes_hit_set_archive":13882,"num_flush":3617,"num_flush_kb":10535558,"num_evict":3241,"num_evict_kb":9431050,"num_promote":4012,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":4,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22590,"ondisk_log_size":22590,"up":8,"acting":8,"num_store_stats":0},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":2,"num_pg":4,"stat_sum":{"num_bytes":29735319058,"num_objects":9940,"num_object_clones":0,"num_object_copies":19880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9940,"num_whiteouts":0,"num_read":15022,"num_read_kb":1498073,"num_write":33027,"num_write_kb":19305998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26626,"ondisk_log_size":26626,"up":8,"acting":8,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640298,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":9873360,"kb_used_data":83864652,"kb_used_omap":12353,"kb_used_meta":0,"kb_avail":83826020,"statfs":{"total":95948165120,"available":85837844480,"internally_reserved":39559168,"allocated":85877403648,"data_stored":85877403648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12649818,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":3,"commit_latency_ns":2000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.505,"15min":0.51500000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.82499999999999996,"5min":1.171,"15min":1.171},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.437,"5min":0.49399999999999999,"15min":0.50800000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.89900000000000002}]},{"osd":1,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.54300000000000004,"15min":0.56799999999999995},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.88600000000000001,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.52300000000000002,"15min":0.54100000000000004},"min":{"1min":0.28599999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.85399999999999998,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.65300000000000002}]},{"osd":2,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.58399999999999996,"15min":0.55500000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":2.681,"5min":2.681,"15min":2.681},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.622,"15min":0.60999999999999999},"min":{"1min":0.32000000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":4.1689999999999996,"5min":4.1689999999999996,"15min":4.1689999999999996},"last":0.98699999999999999}]},{"osd":3,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.502,"15min":0.51700000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.46800000000000003,"15min":0.48599999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.80700000000000005}]},{"osd":4,"last update":"Sun Mar 14 04:57:08 2021","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.52500000000000002,"15min":0.54500000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.89400000000000002,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.872},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.51700000000000002,"15min":0.53300000000000003},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":1.845,"5min":1.845,"15min":1.845},"last":0.83499999999999996}]}]},{"osd":4,"up_from":11,"seq":47244640299,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":488804,"kb_used_data":93226564,"kb_used_omap":12172,"kb_used_meta":0,"kb_avail":93210576,"statfs":{"total":95948165120,"available":95447629824,"internally_reserved":16371712,"allocated":95464001536,"data_stored":95464001536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12464664,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":2,"commit_latency_ns":1000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.505,"15min":0.54900000000000004},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.69399999999999995,"5min":1.036,"15min":1.036},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.53000000000000003,"15min":0.55400000000000005},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.72399999999999998}]},{"osd":1,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.9790000000000001,"5min":1.9790000000000001,"15min":1.9790000000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.53600000000000003,"15min":0.57099999999999995},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.72699999999999998,"5min":1.075,"15min":1.075},"last":0.75600000000000001}]},{"osd":2,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.52500000000000002,"15min":0.56499999999999995},"min":{"1min":0.314,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.72699999999999998,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.58299999999999996,"15min":0.629},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.91100000000000003,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.56399999999999995}]},{"osd":3,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.46400000000000002,"15min":0.50900000000000001},"min":{"1min":0.23699999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.52600000000000002,"5min":1.202,"15min":1.202},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.48799999999999999,"15min":0.52100000000000002},"min":{"1min":0.246,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.55900000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.629}]},{"osd":5,"last update":"Sun Mar 14 04:57:11 2021","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.45100000000000001,"15min":0.47299999999999998},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.81299999999999994,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.5,"15min":0.54400000000000004},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.59699999999999998,"5min":1.121,"15min":1.121},"last":0.76500000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640299,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":4952008,"kb_used_data":88768532,"kb_used_omap":3548,"kb_used_meta":0,"kb_avail":88747372,"statfs":{"total":95948165120,"available":90877308928,"internally_reserved":21667840,"allocated":90898976768,"data_stored":90898976768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3633814,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.54700000000000004,"15min":0.59899999999999998},"min":{"1min":0.253,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.89700000000000002,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":1},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.505,"15min":0.53200000000000003},"min":{"1min":0.35299999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.91600000000000004,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":1.2270000000000001}]},{"osd":1,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.52200000000000002,"15min":0.55400000000000005},"min":{"1min":0.27400000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.77800000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.505,"5min":0.53500000000000003,"15min":0.56499999999999995},"min":{"1min":0.29299999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.93200000000000005}]},{"osd":2,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.54000000000000004,"15min":0.57799999999999996},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.68500000000000005,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.55000000000000004,"15min":0.57199999999999995},"min":{"1min":0.34999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.56399999999999995}]},{"osd":4,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.46600000000000003,"15min":0.51100000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.53300000000000003,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":1.1859999999999999},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.47099999999999997,"15min":0.52300000000000002},"min":{"1min":0.247,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.57499999999999996,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.65600000000000003}]},{"osd":5,"last update":"Sun Mar 14 04:57:13 2021","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.46100000000000002,"15min":0.49199999999999999},"min":{"1min":0.24299999999999999,"5min":0.215,"15min":0.215},"max":{"1min":0.65600000000000003,"5min":0.874,"15min":0.874},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.53200000000000003,"15min":0.57999999999999996},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.69599999999999995,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.751}]}]},{"osd":2,"up_from":11,"seq":47244640298,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":10090296,"kb_used_data":83630492,"kb_used_omap":12460,"kb_used_meta":0,"kb_avail":83609084,"statfs":{"total":95948165120,"available":85615702016,"internally_reserved":21921792,"allocated":85637623808,"data_stored":85637623808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12759532,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":4,"commit_latency_ns":3000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.52400000000000002,"15min":0.48399999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":10.157999999999999,"5min":10.157999999999999,"15min":10.157999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.51600000000000001,"15min":0.502},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.984,"5min":5.984,"15min":5.984},"last":0.51200000000000001}]},{"osd":1,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.67200000000000004,"15min":0.627},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.2140000000000004,"5min":6.2140000000000004,"15min":6.2140000000000004},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.59299999999999997,"15min":0.56599999999999995},"min":{"1min":0.25600000000000001,"5min":0.219,"15min":0.219},"max":{"1min":3.7749999999999999,"5min":3.7749999999999999,"15min":3.7749999999999999},"last":0.49199999999999999}]},{"osd":3,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.70199999999999996,"15min":0.69499999999999995},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":6.2309999999999999,"5min":6.2309999999999999,"15min":6.2309999999999999},"last":0.434},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.71699999999999997,"15min":0.71899999999999997},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":6.0199999999999996,"5min":6.0199999999999996,"15min":6.0199999999999996},"last":0.45500000000000002}]},{"osd":4,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.71499999999999997,"15min":0.71399999999999997},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":6.3499999999999996,"5min":6.3499999999999996,"15min":6.3499999999999996},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.68899999999999995,"15min":0.66900000000000004},"min":{"1min":0.379,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":6.3040000000000003,"5min":6.3040000000000003,"15min":6.3040000000000003},"last":0.52200000000000002}]},{"osd":5,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.72099999999999997,"15min":0.72299999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":6.0490000000000004,"5min":6.0490000000000004,"15min":6.0490000000000004},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.69599999999999995,"15min":0.72999999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":0.70099999999999996}]}]},{"osd":0,"up_from":11,"seq":47244640298,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":498748,"kb_used_data":93219896,"kb_used_omap":12182,"kb_used_meta":0,"kb_avail":93200632,"statfs":{"total":95948165120,"available":95437447168,"internally_reserved":19726336,"allocated":95457173504,"data_stored":95457173504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12474636,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":3,"commit_latency_ns":1000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.44900000000000001,"15min":0.45900000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.503},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.53400000000000003,"15min":0.54600000000000004},"min":{"1min":0.251,"5min":0.249,"15min":0.249},"max":{"1min":2.2509999999999999,"5min":2.2509999999999999,"15min":2.2509999999999999},"last":0.61399999999999999}]},{"osd":2,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56499999999999995,"15min":0.59299999999999997},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.351,"5min":2.351,"15min":2.351},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.57399999999999995,"15min":0.63400000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.80900000000000005,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.67400000000000004}]},{"osd":3,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.58899999999999997,"15min":0.60699999999999998},"min":{"1min":0.31900000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":2.4769999999999999,"5min":2.4769999999999999,"15min":2.4769999999999999},"last":0.746},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.64400000000000002,"15min":0.68999999999999995},"min":{"1min":0.36499999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71399999999999997,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.71099999999999997}]},{"osd":4,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.65400000000000003,"15min":0.71699999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71999999999999997,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.70299999999999996,"15min":0.73599999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":2.4409999999999998,"5min":2.4409999999999998,"15min":2.4409999999999998},"last":0.78700000000000003}]},{"osd":5,"last update":"Sun Mar 14 04:57:10 2021","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.68799999999999994,"15min":0.74299999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.78400000000000003,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.73199999999999998,"15min":0.80800000000000005},"min":{"1min":0.38900000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.77700000000000002,"5min":2.1379999999999999,"15min":2.1379999999999999},"last":0.755}]}]},{"osd":1,"up_from":11,"seq":47244640298,"num_pgs":7,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":14837780,"kb_used_data":78875692,"kb_used_omap":21405,"kb_used_meta":0,"kb_avail":78861600,"statfs":{"total":95948165120,"available":80754278400,"internally_reserved":14430208,"allocated":80768708608,"data_stored":80768708608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21918935,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":3,"commit_latency_ns":2000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.47099999999999997,"15min":0.47999999999999998},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":1.0609999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.308},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.56100000000000005,"15min":0.61299999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.806,"5min":1.806,"15min":1.806},"last":0.34300000000000003}]},{"osd":2,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.501,"15min":0.52800000000000002},"min":{"1min":0.24299999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.60099999999999998,"15min":0.61799999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":2.2570000000000001,"5min":2.2570000000000001,"15min":2.2570000000000001},"last":0.35899999999999999}]},{"osd":3,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.63,"15min":0.69499999999999995},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.0740000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.499},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.621,"15min":0.65500000000000003},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.069,"5min":1.127,"15min":1.127},"last":0.60199999999999998}]},{"osd":4,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.68000000000000005,"15min":0.70399999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.68200000000000005,"15min":0.73799999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.073,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.745}]},{"osd":5,"last update":"Sun Mar 14 04:57:12 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.68899999999999995,"15min":0.74099999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.73899999999999999,"15min":0.73499999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.1669999999999998,"5min":4.1669999999999998,"15min":4.1669999999999998},"last":0.61499999999999999}]}]}]}}

2021-03-14T15:57:02.466 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T04:58:51.516 INFO:teuthology.orchestra.run:waiting for 300
, 2021-03-14T04:58:51.575 INFO:tasks.ceph.mon.a:Stopped
, 2021-03-14T04:58:51.575 INFO:tasks.ceph:Checking cluster log for badness...

2021-03-14T15:57:02.466 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T04:59:10.571 INFO:teuthology.task.internal.syslog:Checking logs for errors...
, 2021-03-14T04:59:10.572 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com
, 2021-03-14T04:59:10.572 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1

2021-03-14T15:57:02.467 WARNING:scrape:Ignoring malparsed backtrace:     dump_program = dump_out.split("from '")[1].split(' ')[0]
, IndexError: list index out of range
, 2021-03-14T04:59:35.497 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload

2021-03-14T15:57:07.153 INFO:scrape:Found 8 distinct failure reasons
2021-03-14T15:57:07.155 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi031 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=381387f151822ee4d4927f3fc584ff1af9deb9ce TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'
2021-03-14T15:57:07.155 INFO:scrape:4 jobs: ['5962318', '5962567', '5962479', '5962412']
2021-03-14T15:57:07.156 INFO:scrape:suites intersection: ['task/test_cephadm}']
2021-03-14T15:57:07.156 INFO:scrape:suites union: ['rados/cephadm/workunits/{distro/centos_latest', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_cephadm}']
2021-03-14T15:57:07.157 INFO:scrape:
2021-03-14T15:57:07.157 INFO:scrape:Failure: Command failed on smithi019 with status 5: 'sudo systemctl stop ceph-207c1eb4-846a-11eb-908c-001a4aab830c@mon.smithi019'
2021-03-14T15:57:07.157 INFO:scrape:8 jobs: ['5962344', '5962571', '5962439', '5962498', '5962443', '5962337', '5962602', '5962505']
2021-03-14T15:57:07.158 INFO:scrape:suites intersection: []
2021-03-14T15:57:07.158 INFO:scrape:suites union: ['fixed-2', 'mode/root', 'msgr/async', 'msgr/async-v1only', 'rados/cephadm/smoke-roleless/{distro/ubuntu_18.04_podman', 'rados/cephadm/smoke/{distro/ubuntu_18.04_podman', 'rados/cephadm/with-work/{distro/ubuntu_18.04_podman', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'start', 'start}', 'task/test_orch_cli}', 'tasks/rados_api_tests}', 'tasks/rados_python}']
2021-03-14T15:57:07.159 INFO:scrape:
2021-03-14T15:57:07.159 INFO:scrape:Failure: reached maximum tries (180) after waiting for 180 seconds
2021-03-14T15:57:07.160 INFO:scrape:2 jobs: ['5962386', '5962618']
2021-03-14T15:57:07.160 INFO:scrape:suites intersection: ['3-start-upgrade', '4-wait', 'fixed-2}', 'rados/cephadm/upgrade/{1-start']
2021-03-14T15:57:07.161 INFO:scrape:suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro$/{centos_latest}', 'distro$/{rhel_latest}', 'fixed-2}', 'rados/cephadm/upgrade/{1-start']
2021-03-14T15:57:07.161 INFO:scrape:
2021-03-14T15:57:07.162 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base'
2021-03-14T15:57:07.162 INFO:scrape:1 jobs: ['5962572']
2021-03-14T15:57:07.162 INFO:scrape:suites: ['msgr-failures/few', 'msgr/async', 'objectstore/filestore-xfs', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'supported-random-distro$/{ubuntu_latest}}']
2021-03-14T15:57:07.163 INFO:scrape:
2021-03-14T15:57:07.163 INFO:scrape:Failure: {'Failure object was': {'smithi042.front.sepia.ceph.com': {'changed': False, 'results': ['yum-plugin-priorities-1.1.31-52.el7.noarch providing yum-plugin-priorities is already installed', 'yum-plugin-fastestmirror-1.1.31-52.el7.noarch providing yum-plugin-fastestmirror is already installed', 'redhat-lsb-4.1-27.el7.x86_64 providing redhat-lsb is already installed', 'sysstat-10.1.5-18.el7.x86_64 providing sysstat is already installed', 'gdb-7.6.1-115.el7.x86_64 providing gdb is already installed', 'git-all-1.8.3.1-20.el7.noarch providing git-all is already installed', 'python-configobj-4.7.2-7.el7.noarch providing python-configobj is already installed', 'gcc-c++-4.8.5-39.el7.x86_64 providing gcc-c++ is already installed', 'libedit-3.0-12.20121213cvs.el7.x86_64 providing libedit is already installed', 'openssl098e-0.9.8e-29.el7_2.3.x86_64 providing openssl098e is already installed', 'boost-thread-1.53.0-27.el7.x86_64 providing boost-thread is already installed', 'xfsprogs-4.5.0-20.el7.x86_64 providing xfsprogs is already installed', 'xfsprogs-devel-4.5.0-20.el7.x86_64 providing xfsprogs-devel is already installed', 'gdisk-0.8.10-3.el7.x86_64 providing gdisk is already installed', 'parted-3.1-31.el7.x86_64 providing parted is already installed', 'libgcrypt-1.5.3-14.el7.x86_64 providing libgcrypt is already installed', 'fuse-2.9.2-11.el7.x86_64 providing fuse is already installed', 'fuse-libs-2.9.2-11.el7.x86_64 providing fuse-libs is already installed', '7:lvm2-2.02.185-2.el7.x86_64 providing lvm2 is already installed', '1:openssl-1.0.2k-19.el7.x86_64 providing openssl is already installed', 'libuuid-2.23.2-61.el7_7.1.x86_64 providing libuuid is already installed', 'btrfs-progs-4.9.1-1.el7.x86_64 providing btrfs-progs is already installed', 'attr-2.4.46-13.el7.x86_64 providing attr is already installed', '1:valgrind-3.14.0-16.el7.x86_64 providing valgrind is already installed', 'python-nose-1.3.7-1.el7.noarch providing python-nose is already installed', 'mpich-3.0-3.0.4-10.el7.x86_64 providing mpich is already installed', 'ant-1.9.4-2.el7.noarch providing ant is already installed', 'lsof-4.87-6.el7.x86_64 providing lsof is already installed', 'iozone-3.424-2_ceph.el7.x86_64 providing iozone is already installed', 'libtool-2.4.2-22.el7_3.x86_64 providing libtool is already installed', 'automake-1.13.4-3.el7.noarch providing automake is already installed', 'gettext-0.19.8.1-2.el7.x86_64 providing gettext is already installed', 'libuuid-devel-2.23.2-61.el7_7.1.x86_64 providing libuuid-devel is already installed', 'libacl-devel-2.2.51-14.el7.x86_64 providing libacl-devel is already installed', 'bc-1.06.95-13.el7.x86_64 providing bc is already installed', 'xfsdump-3.1.7-1.el7.x86_64 providing xfsdump is already installed', 'blktrace-1.0.5-9.el7.x86_64 providing blktrace is already installed', '1:numpy-1.7.1-13.el7.x86_64 providing numpy is already installed', 'python-matplotlib-1.2.0-16.el7.x86_64 providing python-matplotlib is already installed', "No package matching 'qemu-kvm' found available, installed or updated"], 'rc': 126, 'invocation': {'module_args': {'lock_timeout': 30, 'update_cache': False, 'disable_excludes': 'None', 'exclude': [], 'allow_downgrade': False, 'disable_gpg_check': False, 'conf_file': 'None', 'use_backend': 'auto', 'state': 'present', 'disablerepo': [], 'releasever': 'None', 'skip_broken': False, 'autoremove': False, 'download_dir': 'None', 'enable_plugin': [], 'installroot': '/', 'install_weak_deps': True, 'name': ['@core', '@base', 'yum-plugin-priorities', 'yum-plugin-fastestmirror', 'redhat-lsb', 'sysstat', 'gdb', 'git-all', 'python-configobj', 'gcc-c++', 'libedit', 'openssl098e', 'boost-thread', 'xfsprogs', 'xfsprogs-devel', 'gdisk', 'parted', 'libgcrypt', 'fuse', 'fuse-libs', 'lvm2', 'openssl', 'libuuid', 'btrfs-progs', 'attr', 'valgrind', 'python-nose', 'mpich', 'ant', 'lsof', 'iozone', 'libtool', 'automake', 'gettext', 'libuuid-devel', 'libacl-devel', 'bc', 'xfsdump', 'blktrace', 'numpy', 'python-matplotlib', 'qemu-kvm', 'usbredir', 'genisoimage', 'httpd', 'httpd-devel', 'httpd-tools', 'mod_ssl', 'mod_fastcgi-2.4.7-1.ceph.el7', 'libev-devel', 'perl-XML-Twig', 'java-1.6.0-openjdk-devel', 'junit4', 'nfs-utils', 'ncurses-devel', 'python-devel', 'python-virtualenv', 'perl-CPAN', 'podman', 'podman-docker', 'python3'], 'download_only': False, 'bugfix': False, 'list': 'None', 'install_repoquery': True, 'update_only': False, 'disable_plugin': [], 'enablerepo': [], 'security': False, 'validate_certs': True}}, 'msg': "No package matching 'qemu-kvm' found available, installed or updated", '_ansible_no_log': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', 'changed')"}
2021-03-14T15:57:07.164 INFO:scrape:5 jobs: ['5962426', '5962430', '5962586', '5962542', '5962590']
2021-03-14T15:57:07.164 INFO:scrape:suites intersection: []
2021-03-14T15:57:07.165 INFO:scrape:suites union: ['2-repo_digest/defaut', '3-start-upgrade', '4-wait', 'distro$/{rhel_7}', 'fixed-2', 'fixed-2}', 'rados/cephadm/smoke-roleless/{distro/rhel_7', 'rados/cephadm/smoke/{distro/rhel_7', 'rados/cephadm/upgrade/{1-start', 'start}']
2021-03-14T15:57:07.165 INFO:scrape:
2021-03-14T15:57:07.166 INFO:scrape:Failure: Test failure: test_osd_came_back (tasks.mgr.test_progress.TestProgress)
2021-03-14T15:57:07.166 INFO:scrape:2 jobs: ['5962385', '5962599']
2021-03-14T15:57:07.167 INFO:scrape:suites intersection: ['debug/mgr', 'rados/mgr/{clusters/{2-node-mgr}', 'supported-random-distro$/{rhel_latest}', 'tasks/progress}']
2021-03-14T15:57:07.168 INFO:scrape:suites union: ['debug/mgr', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'rados/mgr/{clusters/{2-node-mgr}', 'supported-random-distro$/{rhel_latest}', 'tasks/progress}']
2021-03-14T15:57:07.169 INFO:scrape:
2021-03-14T15:57:07.170 INFO:scrape:Dead: b'2021-03-14T15:53:09.861 DEBUG:teuthology.exit:Finished running handlers'
2021-03-14T15:57:07.170 INFO:scrape:1 jobs: ['5962495']
2021-03-14T15:57:07.170 INFO:scrape:suites: ['cluster/{12-osds', 'msgr-failures/few', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{rhel_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}']
2021-03-14T15:57:07.171 INFO:scrape:
2021-03-14T15:57:07.171 INFO:scrape:Failure: Command failed (workunit test cephadm/test_adoption.sh) on smithi143 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=381387f151822ee4d4927f3fc584ff1af9deb9ce TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh'
2021-03-14T15:57:07.172 INFO:scrape:2 jobs: ['5962620', '5962461']
2021-03-14T15:57:07.172 INFO:scrape:suites intersection: ['rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_adoption}']
2021-03-14T15:57:07.173 INFO:scrape:suites union: ['rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_adoption}']
2021-03-14T15:57:07.173 INFO:scrape:
2021-03-14T15:57:07.185 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/
failed:  19
dead:    6
running: 0
waiting: 0
queued:  0
passed:  295


Fail
=================================================================
[5962318]  rados/cephadm/workunits/{distro/centos_latest task/test_cephadm}
-----------------------------------------------------------------
time:   00:09:12
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962318/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962318/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c4689e1eb20749d49bb11ba3141a70d4

    Command failed (workunit test cephadm/test_cephadm.sh) on smithi031 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=381387f151822ee4d4927f3fc584ff1af9deb9ce
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'

[5962337]  rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 start}
-----------------------------------------------------------------
time:   00:05:39
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962337/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962337/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dcf5023aede64f82922e283fecb15944

    Command failed on smithi041 with status 5: 'sudo systemctl stop
    ceph-7466b04e-8469-11eb-908c-001a4aab830c@mon.a'

[5962344]  rados/cephadm/smoke-roleless/{distro/ubuntu_18.04_podman start}
-----------------------------------------------------------------
time:   00:04:38
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962344/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962344/
sentry: https://sentry.ceph.com/organizations/ceph/?query=05e7af1d70044f4280416c12b97ebb6b

    Command failed on smithi019 with status 5: 'sudo systemctl stop
    ceph-207c1eb4-846a-11eb-908c-001a4aab830c@mon.smithi019'

[5962385]  rados/mgr/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-hybrid supported-random-distro$/{rhel_latest} tasks/progress}
-----------------------------------------------------------------
time:   00:11:22
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962385/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962385/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9ca8acc3da3c4865b8b180cb93d8c099

    Test failure: test_osd_came_back (tasks.mgr.test_progress.TestProgress)

[5962386]  rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{centos_latest} fixed-2}
-----------------------------------------------------------------
time:   00:17:10
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962386/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962386/
sentry: https://sentry.ceph.com/organizations/ceph/?query=101b3cfc44154370b36437448ecebdf8

    reached maximum tries (180) after waiting for 180 seconds

[5962412]  rados/cephadm/workunits/{distro/ubuntu_18.04_podman task/test_cephadm}
-----------------------------------------------------------------
time:   00:06:09
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962412/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962412/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6cbb538631e24f44bc9017a720266756

    Command failed (workunit test cephadm/test_cephadm.sh) on smithi040 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=381387f151822ee4d4927f3fc584ff1af9deb9ce
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'

[5962439]  rados/cephadm/with-work/{distro/ubuntu_18.04_podman fixed-2 mode/root msgr/async-v1only start tasks/rados_python}
-----------------------------------------------------------------
time:   00:07:15
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962439/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962439/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e11f041ad29b41638c5a60bcef3186ba

    Command failed on smithi161 with status 5: 'sudo systemctl stop
    ceph-730fdd5e-8470-11eb-908c-001a4aab830c@mon.a'

[5962443]  rados/cephadm/workunits/{distro/ubuntu_18.04_podman task/test_orch_cli}
-----------------------------------------------------------------
time:   00:05:40
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962443/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962443/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d8d6134ea4ad43dbbb609fa125a3bd5c

    Command failed on smithi058 with status 5: 'sudo systemctl stop
    ceph-4a8704ac-8470-11eb-908c-001a4aab830c@mon.a'

[5962461]  rados/cephadm/workunits/{distro/ubuntu_18.04_podman task/test_adoption}
-----------------------------------------------------------------
time:   00:06:30
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962461/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962461/
sentry: https://sentry.ceph.com/organizations/ceph/?query=625ceefaa6f5483db61eeb967055b6df

    Command failed (workunit test cephadm/test_adoption.sh) on smithi162 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=381387f151822ee4d4927f3fc584ff1af9deb9ce
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh'

[5962479]  rados/cephadm/workunits/{distro/centos_latest task/test_cephadm}
-----------------------------------------------------------------
time:   00:08:50
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962479/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962479/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4b075d6ea4a6465cae36edaa71a43ef8

    Command failed (workunit test cephadm/test_cephadm.sh) on smithi174 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=381387f151822ee4d4927f3fc584ff1af9deb9ce
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'

[5962498]  rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 start}
-----------------------------------------------------------------
time:   00:05:07
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962498/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962498/
sentry: https://sentry.ceph.com/organizations/ceph/?query=80563be508fb4a9888b08231d9e3273c

    Command failed on smithi016 with status 5: 'sudo systemctl stop
    ceph-a1656a4e-8479-11eb-908c-001a4aab830c@mon.a'

[5962505]  rados/cephadm/smoke-roleless/{distro/ubuntu_18.04_podman start}
-----------------------------------------------------------------
time:   00:04:28
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962505/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962505/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a64f2d1d2688446dbcfe44dcc2d899b1

    Command failed on smithi111 with status 5: 'sudo systemctl stop
    ceph-c9c8edc2-847d-11eb-908c-001a4aab830c@mon.smithi111'

[5962567]  rados/cephadm/workunits/{distro/ubuntu_18.04_podman task/test_cephadm}
-----------------------------------------------------------------
time:   00:06:04
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962567/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962567/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bb30719674264a28ab1345aea0bbe5b8

    Command failed (workunit test cephadm/test_cephadm.sh) on smithi099 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=381387f151822ee4d4927f3fc584ff1af9deb9ce
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'

[5962571]  rados/cephadm/with-work/{distro/ubuntu_18.04_podman fixed-2 mode/root msgr/async start tasks/rados_api_tests}
-----------------------------------------------------------------
time:   00:07:33
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962571/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962571/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4ee16e655a384fbb8f31f49e561e04df

    Command failed on smithi041 with status 5: 'sudo systemctl stop
    ceph-9575bfce-8481-11eb-908c-001a4aab830c@mon.a'

[5962572]  rados/singleton/{all/thrash_cache_writeback_proxy_none msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:10:55
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962572/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962572/

    Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000
    --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000
    --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op
    delete 50 --op copy_from 50 --op write_excl 50 --pool base'

[5962599]  rados/mgr/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-comp-zlib supported-random-distro$/{rhel_latest} tasks/progress}
-----------------------------------------------------------------
time:   00:11:19
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962599/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962599/
sentry: https://sentry.ceph.com/organizations/ceph/?query=01008e38102a41ff85f50562b8a3e763

    Test failure: test_osd_came_back (tasks.mgr.test_progress.TestProgress)

[5962602]  rados/cephadm/workunits/{distro/ubuntu_18.04_podman task/test_orch_cli}
-----------------------------------------------------------------
time:   00:05:34
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962602/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962602/
sentry: https://sentry.ceph.com/organizations/ceph/?query=192f07d0256247f2bef440cb09946ecf

    Command failed on smithi038 with status 5: 'sudo systemctl stop
    ceph-b00f7ec8-8482-11eb-908c-001a4aab830c@mon.a'

[5962618]  rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{rhel_latest} fixed-2}
-----------------------------------------------------------------
time:   00:17:17
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962618/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962618/
sentry: https://sentry.ceph.com/organizations/ceph/?query=065a3c32e29e4d3eba3ac02412d4e80e

    reached maximum tries (180) after waiting for 180 seconds

[5962620]  rados/cephadm/workunits/{distro/ubuntu_18.04_podman task/test_adoption}
-----------------------------------------------------------------
time:   00:06:05
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962620/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962620/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e2b41633b9f547fcab76b521629f2bb6

    Command failed (workunit test cephadm/test_adoption.sh) on smithi143 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=381387f151822ee4d4927f3fc584ff1af9deb9ce
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh'



Dead
=================================================================
[5962426]  rados/cephadm/smoke/{distro/rhel_7 fixed-2 start}
-----------------------------------------------------------------
time:   00:03:40
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962426/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962426/
sentry: https://sentry.ceph.com/organizations/ceph/?query=39d28a2b4623485da0aeed61509a9796

    {'Failure object was': {'smithi042.front.sepia.ceph.com': {'changed':
    False, 'results': ['yum-plugin-priorities-1.1.31-52.el7.noarch providing
    yum-plugin-priorities is already installed', 'yum-plugin-
    fastestmirror-1.1.31-52.el7.noarch providing yum-plugin-fastestmirror is
    already installed', 'redhat-lsb-4.1-27.el7.x86_64 providing redhat-lsb is
    already installed', 'sysstat-10.1.5-18.el7.x86_64 providing sysstat is
    already installed', 'gdb-7.6.1-115.el7.x86_64 providing gdb is already
    installed', 'git-all-1.8.3.1-20.el7.noarch providing git-all is already
    installed', 'python-configobj-4.7.2-7.el7.noarch providing python-configobj
    is already installed', 'gcc-c++-4.8.5-39.el7.x86_64 providing gcc-c++ is
    already installed', 'libedit-3.0-12.20121213cvs.el7.x86_64 providing
    libedit is already installed', 'openssl098e-0.9.8e-29.el7_2.3.x86_64
    providing openssl098e is already installed', 'boost-
    thread-1.53.0-27.el7.x86_64 providing boost-thread is already installed',
    'xfsprogs-4.5.0-20.el7.x86_64 providing xfsprogs is already installed',
    'xfsprogs-devel-4.5.0-20.el7.x86_64 providing xfsprogs-devel is already
    installed', 'gdisk-0.8.10-3.el7.x86_64 providing gdisk is already
    installed', 'parted-3.1-31.el7.x86_64 providing parted is already
    installed', 'libgcrypt-1.5.3-14.el7.x86_64 providing libgcrypt is already
    installed', 'fuse-2.9.2-11.el7.x86_64 providing fuse is already installed',
    'fuse-libs-2.9.2-11.el7.x86_64 providing fuse-libs is already installed',
    '7:lvm2-2.02.185-2.el7.x86_64 providing lvm2 is already installed',
    '1:openssl-1.0.2k-19.el7.x86_64 providing openssl is already installed',
    'libuuid-2.23.2-61.el7_7.1.x86_64 providing libuuid is already installed',
    'btrfs-progs-4.9.1-1.el7.x86_64 providing btrfs-progs is already
    installed', 'attr-2.4.46-13.el7.x86_64 providing attr is already
    installed', '1:valgrind-3.14.0-16.el7.x86_64 providing valgrind is already
    installed', 'python-nose-1.3.7-1.el7.noarch providing python-nose is
    already installed', 'mpich-3.0-3.0.4-10.el7.x86_64 providing mpich is
    already installed', 'ant-1.9.4-2.el7.noarch providing ant is already
    installed', 'lsof-4.87-6.el7.x86_64 providing lsof is already installed',
    'iozone-3.424-2_ceph.el7.x86_64 providing iozone is already installed',
    'libtool-2.4.2-22.el7_3.x86_64 providing libtool is already installed',
    'automake-1.13.4-3.el7.noarch providing automake is already installed',
    'gettext-0.19.8.1-2.el7.x86_64 providing gettext is already installed',
    'libuuid-devel-2.23.2-61.el7_7.1.x86_64 providing libuuid-devel is already
    installed', 'libacl-devel-2.2.51-14.el7.x86_64 providing libacl-devel is
    already installed', 'bc-1.06.95-13.el7.x86_64 providing bc is already
    installed', 'xfsdump-3.1.7-1.el7.x86_64 providing xfsdump is already
    installed', 'blktrace-1.0.5-9.el7.x86_64 providing blktrace is already
    installed', '1:numpy-1.7.1-13.el7.x86_64 providing numpy is already
    installed', 'python-matplotlib-1.2.0-16.el7.x86_64 providing python-
    matplotlib is already installed', "No package matching 'qemu-kvm' found
    available, installed or updated"], 'rc': 126, 'invocation': {'module_args':
    {'lock_timeout': 30, 'update_cache': False, 'disable_excludes': 'None',
    'exclude': [], 'allow_downgrade': False, 'disable_gpg_check': False,
    'conf_file': 'None', 'use_backend': 'auto', 'state': 'present',
    'disablerepo': [], 'releasever': 'None', 'skip_broken': False,
    'autoremove': False, 'download_dir': 'None', 'enable_plugin': [],
    'installroot': '/', 'install_weak_deps': True, 'name': ['@core', '@base',
    'yum-plugin-priorities', 'yum-plugin-fastestmirror', 'redhat-lsb',
    'sysstat', 'gdb', 'git-all', 'python-configobj', 'gcc-c++', 'libedit',
    'openssl098e', 'boost-thread', 'xfsprogs', 'xfsprogs-devel', 'gdisk',
    'parted', 'libgcrypt', 'fuse', 'fuse-libs', 'lvm2', 'openssl', 'libuuid',
    'btrfs-progs', 'attr', 'valgrind', 'python-nose', 'mpich', 'ant', 'lsof',
    'iozone', 'libtool', 'automake', 'gettext', 'libuuid-devel', 'libacl-
    devel', 'bc', 'xfsdump', 'blktrace', 'numpy', 'python-matplotlib', 'qemu-
    kvm', 'usbredir', 'genisoimage', 'httpd', 'httpd-devel', 'httpd-tools',
    'mod_ssl', 'mod_fastcgi-2.4.7-1.ceph.el7', 'libev-devel', 'perl-XML-Twig',
    'java-1.6.0-openjdk-devel', 'junit4', 'nfs-utils', 'ncurses-devel',
    'python-devel', 'python-virtualenv', 'perl-CPAN', 'podman', 'podman-
    docker', 'python3'], 'download_only': False, 'bugfix': False, 'list':
    'None', 'install_repoquery': True, 'update_only': False, 'disable_plugin':
    [], 'enablerepo': [], 'security': False, 'validate_certs': True}}, 'msg':
    "No package matching 'qemu-kvm' found available, installed or updated",
    '_ansible_no_log': False}}, 'Traceback (most recent call last)': 'File
    "/home/teuthworker/src/git.ceph.com_git_ceph-cm-
    ansible_master/callback_plugins/failure_log.py", line 44, in log_failure
    log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com
    _git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/pyt
    hon3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return
    dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker
    /src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/v
    irtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in
    dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_gi
    t_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python
    3.6/site-packages/yaml/representer.py", line 27, in represent node =
    self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teut
    hology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/si
    te-packages/yaml/representer.py", line 48, in represent_data node =
    self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s
    rc/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/vir
    tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in
    represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\',
    data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15e
    ae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-
    packages/yaml/representer.py", line 118, in represent_mapping node_value =
    self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi
    t_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python
    3.6/site-packages/yaml/representer.py", line 48, in represent_data node =
    self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s
    rc/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/vir
    tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in
    represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\',
    data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15e
    ae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-
    packages/yaml/representer.py", line 117, in represent_mapping node_key =
    self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_
    teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.
    6/site-packages/yaml/representer.py", line 58, in represent_data node =
    self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ce
    ph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/l
    ib/python3.6/site-packages/yaml/representer.py", line 231, in
    represent_undefined raise RepresenterError("cannot represent an object",
    data)', 'yaml.representer.RepresenterError': "('cannot represent an
    object', 'changed')"}

[5962430]  rados/cephadm/smoke-roleless/{distro/rhel_7 start}
-----------------------------------------------------------------
time:   00:03:47
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962430/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962430/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1afb42d034b64f87abac0a3877e96177

    {'Failure object was': {'smithi119.front.sepia.ceph.com': {'changed':
    False, 'results': ['yum-plugin-priorities-1.1.31-52.el7.noarch providing
    yum-plugin-priorities is already installed', 'yum-plugin-
    fastestmirror-1.1.31-52.el7.noarch providing yum-plugin-fastestmirror is
    already installed', 'redhat-lsb-4.1-27.el7.x86_64 providing redhat-lsb is
    already installed', 'sysstat-10.1.5-18.el7.x86_64 providing sysstat is
    already installed', 'gdb-7.6.1-115.el7.x86_64 providing gdb is already
    installed', 'git-all-1.8.3.1-20.el7.noarch providing git-all is already
    installed', 'python-configobj-4.7.2-7.el7.noarch providing python-configobj
    is already installed', 'gcc-c++-4.8.5-39.el7.x86_64 providing gcc-c++ is
    already installed', 'libedit-3.0-12.20121213cvs.el7.x86_64 providing
    libedit is already installed', 'openssl098e-0.9.8e-29.el7_2.3.x86_64
    providing openssl098e is already installed', 'boost-
    thread-1.53.0-27.el7.x86_64 providing boost-thread is already installed',
    'xfsprogs-4.5.0-20.el7.x86_64 providing xfsprogs is already installed',
    'xfsprogs-devel-4.5.0-20.el7.x86_64 providing xfsprogs-devel is already
    installed', 'gdisk-0.8.10-3.el7.x86_64 providing gdisk is already
    installed', 'parted-3.1-31.el7.x86_64 providing parted is already
    installed', 'libgcrypt-1.5.3-14.el7.x86_64 providing libgcrypt is already
    installed', 'fuse-2.9.2-11.el7.x86_64 providing fuse is already installed',
    'fuse-libs-2.9.2-11.el7.x86_64 providing fuse-libs is already installed',
    '7:lvm2-2.02.185-2.el7.x86_64 providing lvm2 is already installed',
    '1:openssl-1.0.2k-19.el7.x86_64 providing openssl is already installed',
    'libuuid-2.23.2-61.el7_7.1.x86_64 providing libuuid is already installed',
    'btrfs-progs-4.9.1-1.el7.x86_64 providing btrfs-progs is already
    installed', 'attr-2.4.46-13.el7.x86_64 providing attr is already
    installed', '1:valgrind-3.14.0-16.el7.x86_64 providing valgrind is already
    installed', 'python-nose-1.3.7-1.el7.noarch providing python-nose is
    already installed', 'mpich-3.0-3.0.4-10.el7.x86_64 providing mpich is
    already installed', 'ant-1.9.4-2.el7.noarch providing ant is already
    installed', 'lsof-4.87-6.el7.x86_64 providing lsof is already installed',
    'iozone-3.424-2_ceph.el7.x86_64 providing iozone is already installed',
    'libtool-2.4.2-22.el7_3.x86_64 providing libtool is already installed',
    'automake-1.13.4-3.el7.noarch providing automake is already installed',
    'gettext-0.19.8.1-2.el7.x86_64 providing gettext is already installed',
    'libuuid-devel-2.23.2-61.el7_7.1.x86_64 providing libuuid-devel is already
    installed', 'libacl-devel-2.2.51-14.el7.x86_64 providing libacl-devel is
    already installed', 'bc-1.06.95-13.el7.x86_64 providing bc is already
    installed', 'xfsdump-3.1.7-1.el7.x86_64 providing xfsdump is already
    installed', 'blktrace-1.0.5-9.el7.x86_64 providing blktrace is already
    installed', '1:numpy-1.7.1-13.el7.x86_64 providing numpy is already
    installed', 'python-matplotlib-1.2.0-16.el7.x86_64 providing python-
    matplotlib is already installed', "No package matching 'qemu-kvm' found
    available, installed or updated"], 'rc': 126, 'invocation': {'module_args':
    {'lock_timeout': 30, 'update_cache': False, 'disable_excludes': 'None',
    'exclude': [], 'allow_downgrade': False, 'disable_gpg_check': False,
    'conf_file': 'None', 'use_backend': 'auto', 'state': 'present',
    'disablerepo': [], 'releasever': 'None', 'skip_broken': False,
    'autoremove': False, 'download_dir': 'None', 'enable_plugin': [],
    'installroot': '/', 'install_weak_deps': True, 'name': ['@core', '@base',
    'yum-plugin-priorities', 'yum-plugin-fastestmirror', 'redhat-lsb',
    'sysstat', 'gdb', 'git-all', 'python-configobj', 'gcc-c++', 'libedit',
    'openssl098e', 'boost-thread', 'xfsprogs', 'xfsprogs-devel', 'gdisk',
    'parted', 'libgcrypt', 'fuse', 'fuse-libs', 'lvm2', 'openssl', 'libuuid',
    'btrfs-progs', 'attr', 'valgrind', 'python-nose', 'mpich', 'ant', 'lsof',
    'iozone', 'libtool', 'automake', 'gettext', 'libuuid-devel', 'libacl-
    devel', 'bc', 'xfsdump', 'blktrace', 'numpy', 'python-matplotlib', 'qemu-
    kvm', 'usbredir', 'genisoimage', 'httpd', 'httpd-devel', 'httpd-tools',
    'mod_ssl', 'mod_fastcgi-2.4.7-1.ceph.el7', 'libev-devel', 'perl-XML-Twig',
    'java-1.6.0-openjdk-devel', 'junit4', 'nfs-utils', 'ncurses-devel',
    'python-devel', 'python-virtualenv', 'perl-CPAN', 'podman', 'podman-
    docker', 'python3'], 'download_only': False, 'bugfix': False, 'list':
    'None', 'install_repoquery': True, 'update_only': False, 'disable_plugin':
    [], 'enablerepo': [], 'security': False, 'validate_certs': True}}, 'msg':
    "No package matching 'qemu-kvm' found available, installed or updated",
    '_ansible_no_log': False}}, 'Traceback (most recent call last)': 'File
    "/home/teuthworker/src/git.ceph.com_git_ceph-cm-
    ansible_master/callback_plugins/failure_log.py", line 44, in log_failure
    log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com
    _git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/pyt
    hon3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return
    dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker
    /src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/v
    irtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in
    dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_gi
    t_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python
    3.6/site-packages/yaml/representer.py", line 27, in represent node =
    self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teut
    hology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/si
    te-packages/yaml/representer.py", line 48, in represent_data node =
    self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s
    rc/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/vir
    tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in
    represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\',
    data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15e
    ae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-
    packages/yaml/representer.py", line 118, in represent_mapping node_value =
    self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi
    t_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python
    3.6/site-packages/yaml/representer.py", line 48, in represent_data node =
    self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s
    rc/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/vir
    tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in
    represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\',
    data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15e
    ae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-
    packages/yaml/representer.py", line 117, in represent_mapping node_key =
    self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_
    teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.
    6/site-packages/yaml/representer.py", line 58, in represent_data node =
    self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ce
    ph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/l
    ib/python3.6/site-packages/yaml/representer.py", line 231, in
    represent_undefined raise RepresenterError("cannot represent an object",
    data)', 'yaml.representer.RepresenterError': "('cannot represent an
    object', 'changed')"}

[5962495]  rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962495/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962495/

    hit max job timeout

[5962542]  rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{rhel_7} fixed-2}
-----------------------------------------------------------------
time:   00:03:24
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962542/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962542/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8f8182cf842b4e18886ffce970b1cb9a

    {'Failure object was': {'smithi191.front.sepia.ceph.com': {'changed':
    False, 'results': ['yum-plugin-priorities-1.1.31-52.el7.noarch providing
    yum-plugin-priorities is already installed', 'yum-plugin-
    fastestmirror-1.1.31-52.el7.noarch providing yum-plugin-fastestmirror is
    already installed', 'redhat-lsb-4.1-27.el7.x86_64 providing redhat-lsb is
    already installed', 'sysstat-10.1.5-18.el7.x86_64 providing sysstat is
    already installed', 'gdb-7.6.1-115.el7.x86_64 providing gdb is already
    installed', 'git-all-1.8.3.1-20.el7.noarch providing git-all is already
    installed', 'python-configobj-4.7.2-7.el7.noarch providing python-configobj
    is already installed', 'gcc-c++-4.8.5-39.el7.x86_64 providing gcc-c++ is
    already installed', 'libedit-3.0-12.20121213cvs.el7.x86_64 providing
    libedit is already installed', 'openssl098e-0.9.8e-29.el7_2.3.x86_64
    providing openssl098e is already installed', 'boost-
    thread-1.53.0-27.el7.x86_64 providing boost-thread is already installed',
    'xfsprogs-4.5.0-20.el7.x86_64 providing xfsprogs is already installed',
    'xfsprogs-devel-4.5.0-20.el7.x86_64 providing xfsprogs-devel is already
    installed', 'gdisk-0.8.10-3.el7.x86_64 providing gdisk is already
    installed', 'parted-3.1-31.el7.x86_64 providing parted is already
    installed', 'libgcrypt-1.5.3-14.el7.x86_64 providing libgcrypt is already
    installed', 'fuse-2.9.2-11.el7.x86_64 providing fuse is already installed',
    'fuse-libs-2.9.2-11.el7.x86_64 providing fuse-libs is already installed',
    '7:lvm2-2.02.185-2.el7.x86_64 providing lvm2 is already installed',
    '1:openssl-1.0.2k-19.el7.x86_64 providing openssl is already installed',
    'libuuid-2.23.2-61.el7_7.1.x86_64 providing libuuid is already installed',
    'btrfs-progs-4.9.1-1.el7.x86_64 providing btrfs-progs is already
    installed', 'attr-2.4.46-13.el7.x86_64 providing attr is already
    installed', '1:valgrind-3.14.0-16.el7.x86_64 providing valgrind is already
    installed', 'python-nose-1.3.7-1.el7.noarch providing python-nose is
    already installed', 'mpich-3.0-3.0.4-10.el7.x86_64 providing mpich is
    already installed', 'ant-1.9.4-2.el7.noarch providing ant is already
    installed', 'lsof-4.87-6.el7.x86_64 providing lsof is already installed',
    'iozone-3.424-2_ceph.el7.x86_64 providing iozone is already installed',
    'libtool-2.4.2-22.el7_3.x86_64 providing libtool is already installed',
    'automake-1.13.4-3.el7.noarch providing automake is already installed',
    'gettext-0.19.8.1-2.el7.x86_64 providing gettext is already installed',
    'libuuid-devel-2.23.2-61.el7_7.1.x86_64 providing libuuid-devel is already
    installed', 'libacl-devel-2.2.51-14.el7.x86_64 providing libacl-devel is
    already installed', 'bc-1.06.95-13.el7.x86_64 providing bc is already
    installed', 'xfsdump-3.1.7-1.el7.x86_64 providing xfsdump is already
    installed', 'blktrace-1.0.5-9.el7.x86_64 providing blktrace is already
    installed', '1:numpy-1.7.1-13.el7.x86_64 providing numpy is already
    installed', 'python-matplotlib-1.2.0-16.el7.x86_64 providing python-
    matplotlib is already installed', "No package matching 'qemu-kvm' found
    available, installed or updated"], 'rc': 126, 'invocation': {'module_args':
    {'lock_timeout': 30, 'update_cache': False, 'disable_excludes': 'None',
    'exclude': [], 'allow_downgrade': False, 'disable_gpg_check': False,
    'conf_file': 'None', 'use_backend': 'auto', 'state': 'present',
    'disablerepo': [], 'releasever': 'None', 'skip_broken': False,
    'autoremove': False, 'download_dir': 'None', 'enable_plugin': [],
    'installroot': '/', 'install_weak_deps': True, 'name': ['@core', '@base',
    'yum-plugin-priorities', 'yum-plugin-fastestmirror', 'redhat-lsb',
    'sysstat', 'gdb', 'git-all', 'python-configobj', 'gcc-c++', 'libedit',
    'openssl098e', 'boost-thread', 'xfsprogs', 'xfsprogs-devel', 'gdisk',
    'parted', 'libgcrypt', 'fuse', 'fuse-libs', 'lvm2', 'openssl', 'libuuid',
    'btrfs-progs', 'attr', 'valgrind', 'python-nose', 'mpich', 'ant', 'lsof',
    'iozone', 'libtool', 'automake', 'gettext', 'libuuid-devel', 'libacl-
    devel', 'bc', 'xfsdump', 'blktrace', 'numpy', 'python-matplotlib', 'qemu-
    kvm', 'usbredir', 'genisoimage', 'httpd', 'httpd-devel', 'httpd-tools',
    'mod_ssl', 'mod_fastcgi-2.4.7-1.ceph.el7', 'libev-devel', 'perl-XML-Twig',
    'java-1.6.0-openjdk-devel', 'junit4', 'nfs-utils', 'ncurses-devel',
    'python-devel', 'python-virtualenv', 'perl-CPAN', 'podman', 'podman-
    docker', 'python3'], 'download_only': False, 'bugfix': False, 'list':
    'None', 'install_repoquery': True, 'update_only': False, 'disable_plugin':
    [], 'enablerepo': [], 'security': False, 'validate_certs': True}}, 'msg':
    "No package matching 'qemu-kvm' found available, installed or updated",
    '_ansible_no_log': False}}, 'Traceback (most recent call last)': 'File
    "/home/teuthworker/src/git.ceph.com_git_ceph-cm-
    ansible_master/callback_plugins/failure_log.py", line 44, in log_failure
    log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com
    _git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/pyt
    hon3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return
    dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker
    /src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/v
    irtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in
    dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_gi
    t_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python
    3.6/site-packages/yaml/representer.py", line 27, in represent node =
    self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teut
    hology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/si
    te-packages/yaml/representer.py", line 48, in represent_data node =
    self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s
    rc/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/vir
    tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in
    represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\',
    data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15e
    ae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-
    packages/yaml/representer.py", line 118, in represent_mapping node_value =
    self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi
    t_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python
    3.6/site-packages/yaml/representer.py", line 48, in represent_data node =
    self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s
    rc/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/vir
    tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in
    represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\',
    data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15e
    ae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-
    packages/yaml/representer.py", line 117, in represent_mapping node_key =
    self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_
    teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.
    6/site-packages/yaml/representer.py", line 58, in represent_data node =
    self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ce
    ph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/l
    ib/python3.6/site-packages/yaml/representer.py", line 231, in
    represent_undefined raise RepresenterError("cannot represent an object",
    data)', 'yaml.representer.RepresenterError': "('cannot represent an
    object', 'changed')"}

[5962586]  rados/cephadm/smoke/{distro/rhel_7 fixed-2 start}
-----------------------------------------------------------------
time:   00:03:30
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962586/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962586/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6465038a51a6439cbe471f301e3d2e8c

    {'Failure object was': {'smithi117.front.sepia.ceph.com': {'changed':
    False, 'results': ['yum-plugin-priorities-1.1.31-52.el7.noarch providing
    yum-plugin-priorities is already installed', 'yum-plugin-
    fastestmirror-1.1.31-52.el7.noarch providing yum-plugin-fastestmirror is
    already installed', 'redhat-lsb-4.1-27.el7.x86_64 providing redhat-lsb is
    already installed', 'sysstat-10.1.5-18.el7.x86_64 providing sysstat is
    already installed', 'gdb-7.6.1-115.el7.x86_64 providing gdb is already
    installed', 'git-all-1.8.3.1-20.el7.noarch providing git-all is already
    installed', 'python-configobj-4.7.2-7.el7.noarch providing python-configobj
    is already installed', 'gcc-c++-4.8.5-39.el7.x86_64 providing gcc-c++ is
    already installed', 'libedit-3.0-12.20121213cvs.el7.x86_64 providing
    libedit is already installed', 'openssl098e-0.9.8e-29.el7_2.3.x86_64
    providing openssl098e is already installed', 'boost-
    thread-1.53.0-27.el7.x86_64 providing boost-thread is already installed',
    'xfsprogs-4.5.0-20.el7.x86_64 providing xfsprogs is already installed',
    'xfsprogs-devel-4.5.0-20.el7.x86_64 providing xfsprogs-devel is already
    installed', 'gdisk-0.8.10-3.el7.x86_64 providing gdisk is already
    installed', 'parted-3.1-31.el7.x86_64 providing parted is already
    installed', 'libgcrypt-1.5.3-14.el7.x86_64 providing libgcrypt is already
    installed', 'fuse-2.9.2-11.el7.x86_64 providing fuse is already installed',
    'fuse-libs-2.9.2-11.el7.x86_64 providing fuse-libs is already installed',
    '7:lvm2-2.02.185-2.el7.x86_64 providing lvm2 is already installed',
    '1:openssl-1.0.2k-19.el7.x86_64 providing openssl is already installed',
    'libuuid-2.23.2-61.el7_7.1.x86_64 providing libuuid is already installed',
    'btrfs-progs-4.9.1-1.el7.x86_64 providing btrfs-progs is already
    installed', 'attr-2.4.46-13.el7.x86_64 providing attr is already
    installed', '1:valgrind-3.14.0-16.el7.x86_64 providing valgrind is already
    installed', 'python-nose-1.3.7-1.el7.noarch providing python-nose is
    already installed', 'mpich-3.0-3.0.4-10.el7.x86_64 providing mpich is
    already installed', 'ant-1.9.4-2.el7.noarch providing ant is already
    installed', 'lsof-4.87-6.el7.x86_64 providing lsof is already installed',
    'iozone-3.424-2_ceph.el7.x86_64 providing iozone is already installed',
    'libtool-2.4.2-22.el7_3.x86_64 providing libtool is already installed',
    'automake-1.13.4-3.el7.noarch providing automake is already installed',
    'gettext-0.19.8.1-2.el7.x86_64 providing gettext is already installed',
    'libuuid-devel-2.23.2-61.el7_7.1.x86_64 providing libuuid-devel is already
    installed', 'libacl-devel-2.2.51-14.el7.x86_64 providing libacl-devel is
    already installed', 'bc-1.06.95-13.el7.x86_64 providing bc is already
    installed', 'xfsdump-3.1.7-1.el7.x86_64 providing xfsdump is already
    installed', 'blktrace-1.0.5-9.el7.x86_64 providing blktrace is already
    installed', '1:numpy-1.7.1-13.el7.x86_64 providing numpy is already
    installed', 'python-matplotlib-1.2.0-16.el7.x86_64 providing python-
    matplotlib is already installed', "No package matching 'qemu-kvm' found
    available, installed or updated"], 'rc': 126, 'invocation': {'module_args':
    {'lock_timeout': 30, 'update_cache': False, 'disable_excludes': 'None',
    'exclude': [], 'allow_downgrade': False, 'disable_gpg_check': False,
    'conf_file': 'None', 'use_backend': 'auto', 'state': 'present',
    'disablerepo': [], 'releasever': 'None', 'skip_broken': False,
    'autoremove': False, 'download_dir': 'None', 'enable_plugin': [],
    'installroot': '/', 'install_weak_deps': True, 'name': ['@core', '@base',
    'yum-plugin-priorities', 'yum-plugin-fastestmirror', 'redhat-lsb',
    'sysstat', 'gdb', 'git-all', 'python-configobj', 'gcc-c++', 'libedit',
    'openssl098e', 'boost-thread', 'xfsprogs', 'xfsprogs-devel', 'gdisk',
    'parted', 'libgcrypt', 'fuse', 'fuse-libs', 'lvm2', 'openssl', 'libuuid',
    'btrfs-progs', 'attr', 'valgrind', 'python-nose', 'mpich', 'ant', 'lsof',
    'iozone', 'libtool', 'automake', 'gettext', 'libuuid-devel', 'libacl-
    devel', 'bc', 'xfsdump', 'blktrace', 'numpy', 'python-matplotlib', 'qemu-
    kvm', 'usbredir', 'genisoimage', 'httpd', 'httpd-devel', 'httpd-tools',
    'mod_ssl', 'mod_fastcgi-2.4.7-1.ceph.el7', 'libev-devel', 'perl-XML-Twig',
    'java-1.6.0-openjdk-devel', 'junit4', 'nfs-utils', 'ncurses-devel',
    'python-devel', 'python-virtualenv', 'perl-CPAN', 'podman', 'podman-
    docker', 'python3'], 'download_only': False, 'bugfix': False, 'list':
    'None', 'install_repoquery': True, 'update_only': False, 'disable_plugin':
    [], 'enablerepo': [], 'security': False, 'validate_certs': True}}, 'msg':
    "No package matching 'qemu-kvm' found available, installed or updated",
    '_ansible_no_log': False}}, 'Traceback (most recent call last)': 'File
    "/home/teuthworker/src/git.ceph.com_git_ceph-cm-
    ansible_master/callback_plugins/failure_log.py", line 44, in log_failure
    log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com
    _git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/pyt
    hon3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return
    dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker
    /src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/v
    irtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in
    dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_gi
    t_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python
    3.6/site-packages/yaml/representer.py", line 27, in represent node =
    self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teut
    hology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/si
    te-packages/yaml/representer.py", line 48, in represent_data node =
    self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s
    rc/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/vir
    tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in
    represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\',
    data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15e
    ae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-
    packages/yaml/representer.py", line 118, in represent_mapping node_value =
    self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi
    t_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python
    3.6/site-packages/yaml/representer.py", line 48, in represent_data node =
    self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s
    rc/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/vir
    tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in
    represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\',
    data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15e
    ae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-
    packages/yaml/representer.py", line 117, in represent_mapping node_key =
    self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_
    teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.
    6/site-packages/yaml/representer.py", line 58, in represent_data node =
    self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ce
    ph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/l
    ib/python3.6/site-packages/yaml/representer.py", line 231, in
    represent_undefined raise RepresenterError("cannot represent an object",
    data)', 'yaml.representer.RepresenterError': "('cannot represent an
    object', 'changed')"}

[5962590]  rados/cephadm/smoke-roleless/{distro/rhel_7 start}
-----------------------------------------------------------------
time:   00:03:31
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962590/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962590/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bbdeb6bd5b41448ab3c984e900aaf536

    {'Failure object was': {'smithi017.front.sepia.ceph.com': {'changed':
    False, 'results': ['yum-plugin-priorities-1.1.31-52.el7.noarch providing
    yum-plugin-priorities is already installed', 'yum-plugin-
    fastestmirror-1.1.31-52.el7.noarch providing yum-plugin-fastestmirror is
    already installed', 'redhat-lsb-4.1-27.el7.x86_64 providing redhat-lsb is
    already installed', 'sysstat-10.1.5-18.el7.x86_64 providing sysstat is
    already installed', 'gdb-7.6.1-115.el7.x86_64 providing gdb is already
    installed', 'git-all-1.8.3.1-20.el7.noarch providing git-all is already
    installed', 'python-configobj-4.7.2-7.el7.noarch providing python-configobj
    is already installed', 'gcc-c++-4.8.5-39.el7.x86_64 providing gcc-c++ is
    already installed', 'libedit-3.0-12.20121213cvs.el7.x86_64 providing
    libedit is already installed', 'openssl098e-0.9.8e-29.el7_2.3.x86_64
    providing openssl098e is already installed', 'boost-
    thread-1.53.0-27.el7.x86_64 providing boost-thread is already installed',
    'xfsprogs-4.5.0-20.el7.x86_64 providing xfsprogs is already installed',
    'xfsprogs-devel-4.5.0-20.el7.x86_64 providing xfsprogs-devel is already
    installed', 'gdisk-0.8.10-3.el7.x86_64 providing gdisk is already
    installed', 'parted-3.1-31.el7.x86_64 providing parted is already
    installed', 'libgcrypt-1.5.3-14.el7.x86_64 providing libgcrypt is already
    installed', 'fuse-2.9.2-11.el7.x86_64 providing fuse is already installed',
    'fuse-libs-2.9.2-11.el7.x86_64 providing fuse-libs is already installed',
    '7:lvm2-2.02.185-2.el7.x86_64 providing lvm2 is already installed',
    '1:openssl-1.0.2k-19.el7.x86_64 providing openssl is already installed',
    'libuuid-2.23.2-61.el7_7.1.x86_64 providing libuuid is already installed',
    'btrfs-progs-4.9.1-1.el7.x86_64 providing btrfs-progs is already
    installed', 'attr-2.4.46-13.el7.x86_64 providing attr is already
    installed', '1:valgrind-3.14.0-16.el7.x86_64 providing valgrind is already
    installed', 'python-nose-1.3.7-1.el7.noarch providing python-nose is
    already installed', 'mpich-3.0-3.0.4-10.el7.x86_64 providing mpich is
    already installed', 'ant-1.9.4-2.el7.noarch providing ant is already
    installed', 'lsof-4.87-6.el7.x86_64 providing lsof is already installed',
    'iozone-3.424-2_ceph.el7.x86_64 providing iozone is already installed',
    'libtool-2.4.2-22.el7_3.x86_64 providing libtool is already installed',
    'automake-1.13.4-3.el7.noarch providing automake is already installed',
    'gettext-0.19.8.1-2.el7.x86_64 providing gettext is already installed',
    'libuuid-devel-2.23.2-61.el7_7.1.x86_64 providing libuuid-devel is already
    installed', 'libacl-devel-2.2.51-14.el7.x86_64 providing libacl-devel is
    already installed', 'bc-1.06.95-13.el7.x86_64 providing bc is already
    installed', 'xfsdump-3.1.7-1.el7.x86_64 providing xfsdump is already
    installed', 'blktrace-1.0.5-9.el7.x86_64 providing blktrace is already
    installed', '1:numpy-1.7.1-13.el7.x86_64 providing numpy is already
    installed', 'python-matplotlib-1.2.0-16.el7.x86_64 providing python-
    matplotlib is already installed', "No package matching 'qemu-kvm' found
    available, installed or updated"], 'rc': 126, 'invocation': {'module_args':
    {'lock_timeout': 30, 'update_cache': False, 'disable_excludes': 'None',
    'exclude': [], 'allow_downgrade': False, 'disable_gpg_check': False,
    'conf_file': 'None', 'use_backend': 'auto', 'state': 'present',
    'disablerepo': [], 'releasever': 'None', 'skip_broken': False,
    'autoremove': False, 'download_dir': 'None', 'enable_plugin': [],
    'installroot': '/', 'install_weak_deps': True, 'name': ['@core', '@base',
    'yum-plugin-priorities', 'yum-plugin-fastestmirror', 'redhat-lsb',
    'sysstat', 'gdb', 'git-all', 'python-configobj', 'gcc-c++', 'libedit',
    'openssl098e', 'boost-thread', 'xfsprogs', 'xfsprogs-devel', 'gdisk',
    'parted', 'libgcrypt', 'fuse', 'fuse-libs', 'lvm2', 'openssl', 'libuuid',
    'btrfs-progs', 'attr', 'valgrind', 'python-nose', 'mpich', 'ant', 'lsof',
    'iozone', 'libtool', 'automake', 'gettext', 'libuuid-devel', 'libacl-
    devel', 'bc', 'xfsdump', 'blktrace', 'numpy', 'python-matplotlib', 'qemu-
    kvm', 'usbredir', 'genisoimage', 'httpd', 'httpd-devel', 'httpd-tools',
    'mod_ssl', 'mod_fastcgi-2.4.7-1.ceph.el7', 'libev-devel', 'perl-XML-Twig',
    'java-1.6.0-openjdk-devel', 'junit4', 'nfs-utils', 'ncurses-devel',
    'python-devel', 'python-virtualenv', 'perl-CPAN', 'podman', 'podman-
    docker', 'python3'], 'download_only': False, 'bugfix': False, 'list':
    'None', 'install_repoquery': True, 'update_only': False, 'disable_plugin':
    [], 'enablerepo': [], 'security': False, 'validate_certs': True}}, 'msg':
    "No package matching 'qemu-kvm' found available, installed or updated",
    '_ansible_no_log': False}}, 'Traceback (most recent call last)': 'File
    "/home/teuthworker/src/git.ceph.com_git_ceph-cm-
    ansible_master/callback_plugins/failure_log.py", line 44, in log_failure
    log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com
    _git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/pyt
    hon3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return
    dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker
    /src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/v
    irtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in
    dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_gi
    t_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python
    3.6/site-packages/yaml/representer.py", line 27, in represent node =
    self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teut
    hology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/si
    te-packages/yaml/representer.py", line 48, in represent_data node =
    self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s
    rc/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/vir
    tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in
    represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\',
    data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15e
    ae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-
    packages/yaml/representer.py", line 118, in represent_mapping node_value =
    self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi
    t_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python
    3.6/site-packages/yaml/representer.py", line 48, in represent_data node =
    self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s
    rc/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/vir
    tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in
    represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\',
    data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15e
    ae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-
    packages/yaml/representer.py", line 117, in represent_mapping node_key =
    self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_
    teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.
    6/site-packages/yaml/representer.py", line 58, in represent_data node =
    self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ce
    ph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/l
    ib/python3.6/site-packages/yaml/representer.py", line 231, in
    represent_undefined raise RepresenterError("cannot represent an object",
    data)', 'yaml.representer.RepresenterError': "('cannot represent an
    object', 'changed')"}



Pass
=================================================================
[5962316] rados/singleton/{all/dump-stuck msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}}
time:   00:08:28
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962316/

[5962317] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/off msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_latest} thrashers/morepggrow thrashosds-health workloads/cache}
time:   00:17:42
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962317/

[5962319] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects}
time:   00:21:45
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962319/

[5962320] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 start}
time:   00:13:21
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962320/

[5962321] rados/mgr/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-comp-zlib supported-random-distro$/{centos_latest} tasks/insights}
time:   00:11:53
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962321/

[5962322] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub}
time:   01:28:23
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962322/

[5962323] rados/singleton/{all/ec-lost-unfound msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}}
time:   00:28:39
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962323/

[5962324] rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} tasks/rados_python}
time:   00:13:27
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962324/

[5962325] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04 start}
time:   00:11:59
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962325/

[5962326] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/dedup_tier}
time:   00:09:09
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962326/

[5962327] rados/perf/{ceph objectstore/bluestore-bitmap openstack settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read}
time:   00:09:59
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962327/

[5962328] rados/singleton-nomsgr/{all/recovery-unfound-found rados supported-random-distro$/{rhel_latest}}
time:   00:30:02
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962328/

[5962329] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/packaged msgr/async-v2only start tasks/rados_api_tests}
time:   00:30:07
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962329/

[5962330] rados/dashboard/{clusters/{2-node-mgr} debug/mgr objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} tasks/dashboard}
time:   01:06:28
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962330/

[5962331] rados/objectstore/{backends/objectstore supported-random-distro$/{rhel_latest}}
time:   04:20:55
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962331/

[5962332] rados/singleton-bluestore/{all/cephtool msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}}
time:   00:28:13
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962332/

[5962333] rados/singleton/{all/erasure-code-nonregression msgr-failures/many msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}}
time:   00:05:23
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962333/

[5962334] rados/cephadm/workunits/{distro/ubuntu_18.04_podman task/test_cephadm_repos}
time:   00:04:35
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962334/

[5962335] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/pool-snaps-few-objects}
time:   00:33:32
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962335/

[5962336] rados/monthrash/{ceph clusters/9-mons msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_latest} thrashers/many workloads/pool-create-delete}
time:   00:20:14
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962336/

[5962338] rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v2only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} msgr-failures/fastclose rados thrashers/morepggrow thrashosds-health workloads/test_rbd_api}
time:   00:20:31
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962338/

[5962339] rados/multimon/{clusters/9 msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{rhel_latest} tasks/mon_clock_no_skews}
time:   00:09:20
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962339/

[5962340] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}
time:   00:11:21
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962340/

[5962341] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{rhel_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
time:   00:27:45
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962341/

[5962342] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none msgr-failures/few msgr/async objectstore/bluestore-stupid rados tasks/mon_recovery validater/valgrind}
time:   00:25:50
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962342/

[5962343] rados/singleton/{all/lost-unfound-delete msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}}
time:   00:28:07
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962343/

[5962345] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{rhel_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
time:   00:13:01
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962345/

[5962346] rados/perf/{ceph objectstore/bluestore-comp openstack settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read}
time:   00:10:07
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962346/

[5962347] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/root msgr/async start tasks/rados_python}
time:   00:20:58
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962347/

[5962348] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/off msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/rados_api_tests}
time:   00:20:34
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962348/

[5962349] rados/singleton-nomsgr/{all/version-number-sanity rados supported-random-distro$/{rhel_latest}}
time:   00:10:29
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962349/

[5962350] rados/singleton/{all/lost-unfound msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}}
time:   00:22:51
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962350/

[5962351] rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} tasks/rados_stress_watch}
time:   00:15:08
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962351/

[5962352] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
time:   00:24:52
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962352/

[5962353] rados/mgr/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-comp-zstd supported-random-distro$/{rhel_latest} tasks/module_selftest}
time:   00:31:03
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962353/

[5962354] rados/cephadm/workunits/{distro/centos_latest task/test_orch_cli}
time:   00:17:14
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962354/

[5962355] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 start}
time:   00:13:37
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962355/

[5962356] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast msgr-failures/few rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-snaps-few-objects-overwrites}
time:   00:24:49
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962356/

[5962357] rados/singleton/{all/max-pg-per-osd.from-mon msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}}
time:   00:07:54
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962357/

[5962358] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/radosbench-high-concurrency}
time:   00:29:52
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962358/

[5962359] rados/cephadm/smoke-roleless/{distro/ubuntu_latest start}
time:   00:13:45
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962359/

[5962360] rados/perf/{ceph objectstore/bluestore-low-osd-mem-target openstack settings/optimized ubuntu_latest workloads/radosbench_4M_write}
time:   00:08:35
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962360/

[5962361] rados/cephadm/with-work/{distro/rhel_8.0 fixed-2 mode/packaged msgr/async-v1only start tasks/rados_api_tests}
time:   00:33:58
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962361/

[5962362] rados/dashboard/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-bitmap supported-random-distro$/{rhel_latest} tasks/dashboard}
time:   01:07:52
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962362/

[5962363] rados/objectstore/{backends/alloc-hint supported-random-distro$/{rhel_latest}}
time:   00:09:46
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962363/

[5962364] rados/rest/{mgr-restful supported-random-distro$/{rhel_latest}}
time:   00:10:48
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962364/

[5962365] rados/singleton-nomsgr/{all/admin_socket_output rados supported-random-distro$/{rhel_latest}}
time:   00:18:38
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962365/

[5962366] rados/standalone/{supported-random-distro$/{centos_latest} workloads/crush}
time:   00:09:52
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962366/

[5962367] rados/upgrade/mimic-x-singleton/{0-cluster/{openstack start} 1-install/mimic 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap thrashosds-health ubuntu_latest}
time:   05:33:00
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962367/

[5962368] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest}
time:   00:17:13
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962368/

[5962369] rados/singleton/{all/max-pg-per-osd.from-primary msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_latest}}
time:   00:15:47
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962369/

[5962370] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/radosbench}
time:   01:21:13
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962370/

[5962371] rados/cephadm/orchestrator_cli/{2-node-mgr orchestrator_cli supported-random-distro$/{rhel_latest}}
time:   00:13:52
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962371/

[5962372] rados/monthrash/{ceph clusters/3-mons msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_latest} thrashers/one workloads/rados_5925}
time:   00:10:31
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962372/

[5962373] rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/off distro$/{ubuntu_18.04} msgr-failures/few rados thrashers/none thrashosds-health workloads/cache-snaps}
time:   00:26:09
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962373/

[5962374] rados/multimon/{clusters/21 msgr-failures/many msgr/async-v2only no_pools objectstore/filestore-xfs rados supported-random-distro$/{centos_latest} tasks/mon_clock_with_skews}
time:   00:07:45
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962374/

[5962375] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/fastclose objectstore/filestore-xfs rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2}
time:   00:22:50
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962375/

[5962376] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/fastclose objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
time:   00:28:05
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962376/

[5962377] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados tasks/rados_api_tests validater/lockdep}
time:   00:20:24
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962377/

[5962378] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
time:   00:10:43
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962378/

[5962379] rados/cephadm/smoke/{distro/centos_7 fixed-2 start}
time:   00:15:58
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962379/

[5962380] rados/singleton/{all/max-pg-per-osd.from-replica msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_latest}}
time:   00:14:44
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962380/

[5962381] rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/rados_striper}
time:   00:09:54
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962381/

[5962382] rados/cephadm/smoke-roleless/{distro/centos_7 start}
time:   00:15:11
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962382/

[5962383] rados/perf/{ceph objectstore/bluestore-stupid openstack settings/optimized ubuntu_latest workloads/radosbench_omap_write}
time:   00:20:28
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962383/

[5962384] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/off msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/redirect}
time:   00:11:50
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962384/

[5962387] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1}
time:   00:24:48
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962387/

[5962388] rados/singleton/{all/mon-auth-caps msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_latest}}
time:   00:10:23
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962388/

[5962389] rados/singleton-nomsgr/{all/balancer rados supported-random-distro$/{rhel_latest}}
time:   00:11:03
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962389/

[5962390] rados/cephadm/workunits/{distro/centos_latest task/test_adoption}
time:   00:08:07
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962390/

[5962391] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap msgr-failures/fastclose msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/redirect_promote_tests}
time:   00:10:28
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962391/

[5962392] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/root msgr/async-v2only start tasks/rados_python}
time:   00:22:47
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962392/

[5962393] rados/singleton/{all/mon-config-key-caps msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}}
time:   00:08:46
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962393/

[5962394] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 start}
time:   00:13:47
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962394/

[5962395] rados/perf/{ceph objectstore/bluestore-basic-min-osd-mem-target openstack settings/optimized ubuntu_latest workloads/sample_fio}
time:   00:10:53
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962395/

[5962396] rados/cephadm/smoke-roleless/{distro/centos_8.0 start}
time:   00:13:12
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962396/

[5962397] rados/dashboard/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_latest} tasks/dashboard}
time:   01:09:25
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962397/

[5962398] rados/monthrash/{ceph clusters/9-mons msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/sync-many workloads/rados_api_tests}
time:   00:24:14
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962398/

[5962399] rados/multimon/{clusters/3 msgr-failures/few msgr/async no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery}
time:   00:08:56
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962399/

[5962400] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{rhel_latest}}
time:   00:13:52
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962400/

[5962401] rados/singleton/{all/mon-config-keys msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_latest}}
time:   00:15:07
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962401/

[5962402] rados/singleton-bluestore/{all/cephtool msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}}
time:   00:28:08
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962402/

[5962403] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/redirect_set_object}
time:   00:11:20
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962403/

[5962404] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}
time:   00:13:18
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962404/

[5962405] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
time:   00:24:50
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962405/

[5962406] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal msgr-failures/osd-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites}
time:   00:27:02
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962406/

[5962407] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
time:   00:10:45
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962407/

[5962408] rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/jewel-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/radosbench}
time:   00:54:20
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962408/

[5962409] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind}
time:   00:40:41
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962409/

[5962410] rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_big}
time:   00:28:49
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962410/

[5962411] rados/singleton-nomsgr/{all/cache-fs-trunc rados supported-random-distro$/{ubuntu_latest}}
time:   00:07:11
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962411/

[5962413] rados/cephadm/smoke/{distro/centos_latest fixed-2 start}
time:   00:13:25
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962413/

[5962414] rados/singleton/{all/mon-config msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_latest}}
time:   00:10:20
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962414/

[5962415] rados/mgr/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_latest} tasks/prometheus}
time:   00:11:40
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962415/

[5962416] rados/standalone/{supported-random-distro$/{rhel_latest} workloads/erasure-code}
time:   00:37:03
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962416/

[5962417] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/off msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/set-chunk-promote-flush}
time:   00:09:45
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962417/

[5962418] rados/cephadm/smoke-roleless/{distro/centos_latest start}
time:   00:12:06
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962418/

[5962419] rados/perf/{ceph objectstore/bluestore-bitmap openstack settings/optimized ubuntu_latest workloads/sample_radosbench}
time:   00:08:57
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962419/

[5962420] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1}
time:   00:22:02
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962420/

[5962421] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/packaged msgr/async start tasks/rados_api_tests}
time:   00:30:03
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962421/

[5962422] rados/singleton/{all/osd-backfill msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_latest}}
time:   00:19:54
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962422/

[5962423] rados/cephadm/workunits/{distro/centos_latest task/test_cephadm_repos}
time:   00:05:12
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962423/

[5962424] rados/singleton-nomsgr/{all/ceph-kvstore-tool rados supported-random-distro$/{centos_latest}}
time:   00:07:58
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962424/

[5962425] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/set-chunks-read}
time:   00:15:14
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962425/

[5962427] rados/singleton/{all/osd-recovery-incomplete msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_latest}}
time:   00:23:05
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962427/

[5962428] rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} tasks/rados_workunit_loadgen_mix}
time:   00:19:46
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962428/

[5962429] rados/perf/{ceph objectstore/bluestore-bitmap openstack settings/optimized ubuntu_latest workloads/cosbench_64K_read_write}
time:   00:15:55
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962429/

[5962431] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
time:   00:12:17
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962431/

[5962432] rados/multimon/{clusters/6 msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} tasks/mon_clock_no_skews}
time:   00:07:17
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962432/

[5962433] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2}
time:   00:25:55
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962433/

[5962434] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
time:   00:24:11
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962434/

[5962435] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados tasks/rados_cls_all validater/lockdep}
time:   00:13:21
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962435/

[5962436] rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/jewel backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/off distro$/{ubuntu_18.04} msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/rbd_cls}
time:   00:20:09
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962436/

[5962437] rados/monthrash/{ceph clusters/3-mons msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_mon_osdmap_prune}
time:   00:37:36
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962437/

[5962438] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/small-objects-balanced}
time:   00:20:14
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962438/

[5962440] rados/singleton/{all/osd-recovery msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}}
time:   00:12:12
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962440/

[5962441] rados/dashboard/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-comp-snappy supported-random-distro$/{centos_latest} tasks/dashboard}
time:   01:06:40
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962441/

[5962442] rados/objectstore/{backends/filejournal supported-random-distro$/{centos_latest}}
time:   00:07:41
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962442/

[5962444] rados/singleton-nomsgr/{all/ceph-post-file rados supported-random-distro$/{ubuntu_latest}}
time:   00:05:39
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962444/

[5962445] rados/mgr/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} tasks/workunits}
time:   00:07:02
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962445/

[5962446] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 start}
time:   00:15:34
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962446/

[5962447] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/off msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_latest} thrashers/morepggrow thrashosds-health workloads/small-objects-localized}
time:   00:23:44
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962447/

[5962448] rados/singleton/{all/peer msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}}
time:   00:08:07
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962448/

[5962449] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast msgr-failures/fastclose rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-fast-read-overwrites}
time:   00:21:39
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962449/

[5962450] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal msgr-failures/fastclose objectstore/filestore-xfs rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-radosbench}
time:   01:33:38
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962450/

[5962451] rados/perf/{ceph objectstore/bluestore-comp openstack settings/optimized ubuntu_latest workloads/cosbench_64K_write}
time:   00:17:50
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962451/

[5962452] rados/cephadm/smoke-roleless/{distro/rhel_8.0 start}
time:   00:14:51
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962452/

[5962453] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{ubuntu_latest} fixed-2}
time:   00:18:26
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962453/

[5962454] rados/singleton/{all/pg-autoscaler-progress-off msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}}
time:   00:08:43
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962454/

[5962455] rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/many msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_latest} tasks/rados_workunit_loadgen_mostlyread}
time:   00:24:29
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962455/

[5962456] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/small-objects}
time:   00:17:42
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962456/

[5962457] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/packaged msgr/async-v1only start tasks/rados_python}
time:   00:22:38
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962457/

[5962458] rados/standalone/{supported-random-distro$/{rhel_latest} workloads/mgr}
time:   00:13:14
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962458/

[5962459] rados/singleton-nomsgr/{all/export-after-evict rados supported-random-distro$/{rhel_latest}}
time:   00:09:45
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962459/

[5962460] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
time:   00:13:56
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962460/

[5962462] rados/singleton/{all/pg-autoscaler msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}}
time:   00:10:11
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962462/

[5962463] rados/perf/{ceph objectstore/bluestore-low-osd-mem-target openstack settings/optimized ubuntu_latest workloads/fio_4K_rand_read}
time:   00:11:28
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962463/

[5962464] rados/multimon/{clusters/9 msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_latest} tasks/mon_clock_with_skews}
time:   00:10:18
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962464/

[5962465] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}
time:   00:09:45
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962465/

[5962466] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
time:   00:26:49
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962466/

[5962467] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/mon_recovery validater/valgrind}
time:   00:24:52
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962467/

[5962468] rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} msgr-failures/few rados thrashers/careful thrashosds-health workloads/snaps-few-objects}
time:   00:48:05
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962468/

[5962469] rados/monthrash/{ceph clusters/9-mons msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_latest} thrashers/force-sync-many workloads/rados_mon_workunits}
time:   00:26:15
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962469/

[5962470] rados/cephadm/smoke/{distro/rhel_latest fixed-2 start}
time:   00:13:48
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962470/

[5962471] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-balanced}
time:   00:26:44
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962471/

[5962472] rados/cephadm/smoke-roleless/{distro/rhel_latest start}
time:   00:12:35
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962472/

[5962473] rados/singleton/{all/pg-removal-interruption msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}}
time:   00:07:40
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962473/

[5962474] rados/dashboard/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-comp-zlib supported-random-distro$/{rhel_latest} tasks/dashboard}
time:   01:07:38
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962474/

[5962475] rados/mgr/{clusters/{2-node-mgr} debug/mgr objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} tasks/crash}
time:   00:09:08
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962475/

[5962476] rados/objectstore/{backends/filestore-idempotent-aio-journal supported-random-distro$/{rhel_latest}}
time:   02:29:20
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962476/

[5962477] rados/singleton-bluestore/{all/cephtool msgr-failures/many msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}}
time:   00:28:19
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962477/

[5962478] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest}
time:   00:17:48
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962478/

[5962480] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-balanced}
time:   00:21:08
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962480/

[5962481] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/off msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/snaps-few-objects-localized}
time:   00:25:38
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962481/

[5962482] rados/singleton-nomsgr/{all/full-tiering rados supported-random-distro$/{ubuntu_latest}}
time:   00:07:00
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962482/

[5962483] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/root msgr/async-v2only start tasks/rados_api_tests}
time:   00:29:00
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962483/

[5962484] rados/singleton/{all/radostool msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}}
time:   00:12:33
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962484/

[5962485] rados/perf/{ceph objectstore/bluestore-stupid openstack settings/optimized ubuntu_latest workloads/fio_4K_rand_rw}
time:   00:11:16
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962485/

[5962486] rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_latest} tasks/readwrite}
time:   00:13:38
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962486/

[5962487] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 start}
time:   00:13:51
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962487/

[5962488] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04 start}
time:   00:12:13
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962488/

[5962489] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap msgr-failures/fastclose msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/snaps-few-objects}
time:   00:27:52
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962489/

[5962490] rados/singleton/{all/random-eio msgr-failures/many msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_latest}}
time:   00:19:32
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962490/

[5962491] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal msgr-failures/few rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-overwrites}
time:   00:20:58
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962491/

[5962492] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
time:   00:10:57
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962492/

[5962493] rados/cephadm/workunits/{distro/ubuntu_18.04_podman task/test_cephadm_repos}
time:   00:04:39
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962493/

[5962494] rados/multimon/{clusters/21 msgr-failures/many msgr/async no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery}
time:   00:22:51
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962494/

[5962496] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
time:   00:23:39
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962496/

[5962497] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep}
time:   00:20:12
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962497/

[5962499] rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/off distro$/{ubuntu_18.04} msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/test_rbd_api}
time:   00:19:42
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962499/

[5962500] rados/monthrash/{ceph clusters/3-mons msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_latest} thrashers/many workloads/rados_mon_workunits}
time:   00:29:10
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962500/

[5962501] rados/singleton-nomsgr/{all/health-warnings rados supported-random-distro$/{centos_latest}}
time:   00:08:53
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962501/

[5962502] rados/perf/{ceph objectstore/bluestore-basic-min-osd-mem-target openstack settings/optimized ubuntu_latest workloads/fio_4M_rand_read}
time:   00:08:52
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962502/

[5962503] rados/singleton/{all/rebuild-mondb msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}}
time:   00:12:25
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962503/

[5962504] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/write_fadvise_dontneed}
time:   00:13:06
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962504/

[5962506] rados/mgr/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-bitmap supported-random-distro$/{centos_latest} tasks/failover}
time:   00:11:58
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962506/

[5962507] rados/standalone/{supported-random-distro$/{centos_latest} workloads/misc}
time:   00:23:22
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962507/

[5962508] rados/cephadm/with-work/{distro/rhel_8.0 fixed-2 mode/packaged msgr/async start tasks/rados_python}
time:   00:24:17
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962508/

[5962509] rados/singleton/{all/recovery-preemption msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}}
time:   00:16:11
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962509/

[5962510] rados/dashboard/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-comp-zstd supported-random-distro$/{centos_latest} tasks/dashboard}
time:   01:03:14
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962510/

[5962511] rados/objectstore/{backends/filestore-idempotent supported-random-distro$/{ubuntu_latest}}
time:   02:13:22
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962511/

[5962512] rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/repair_test}
time:   00:14:19
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962512/

[5962513] rados/cephadm/workunits/{distro/centos_latest task/test_orch_cli}
time:   00:16:21
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962513/

[5962514] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read}
time:   00:21:45
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962514/

[5962515] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/off msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/admin_socket_objecter_requests}
time:   00:21:39
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962515/

[5962516] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 start}
time:   00:13:33
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962516/

[5962517] rados/perf/{ceph objectstore/bluestore-bitmap openstack settings/optimized ubuntu_latest workloads/fio_4M_rand_rw}
time:   00:09:04
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962517/

[5962518] rados/singleton/{all/resolve_stuck_peering msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}}
time:   00:08:24
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962518/

[5962519] rados/singleton-nomsgr/{all/large-omap-object-warnings rados supported-random-distro$/{rhel_latest}}
time:   00:10:21
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962519/

[5962520] rados/cephadm/smoke-roleless/{distro/ubuntu_latest start}
time:   00:12:01
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962520/

[5962521] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
time:   00:13:10
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962521/

[5962522] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/cache-agent-big}
time:   00:18:02
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962522/

[5962523] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/root msgr/async-v1only start tasks/rados_api_tests}
time:   00:31:40
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962523/

[5962524] rados/singleton/{all/test-crash msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}}
time:   00:08:02
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962524/

[5962525] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap thrashosds-health ubuntu_latest}
time:   03:17:14
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962525/

[5962526] rados/cephadm/orchestrator_cli/{2-node-mgr orchestrator_cli supported-random-distro$/{centos_latest}}
time:   00:11:18
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962526/

[5962527] rados/multimon/{clusters/3 msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_latest} tasks/mon_clock_no_skews}
time:   00:09:15
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962527/

[5962528] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{rhel_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}
time:   00:12:11
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962528/

[5962529] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
time:   00:23:47
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962529/

[5962530] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados tasks/rados_cls_all validater/valgrind}
time:   01:04:24
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962530/

[5962531] rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/cache-snaps}
time:   00:34:51
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962531/

[5962532] rados/monthrash/{ceph clusters/9-mons msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_latest} thrashers/one workloads/snaps-few-objects}
time:   00:34:06
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962532/

[5962533] rados/cephadm/smoke/{distro/centos_7 fixed-2 start}
time:   00:15:48
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962533/

[5962534] rados/perf/{ceph objectstore/bluestore-comp openstack settings/optimized ubuntu_latest workloads/fio_4M_rand_write}
time:   00:12:53
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962534/

[5962535] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache-agent-small}
time:   00:14:19
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962535/

[5962536] rados/singleton/{all/test_envlibrados_for_rocksdb msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}}
time:   00:12:36
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962536/

[5962537] rados/mgr/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_latest} tasks/insights}
time:   00:14:44
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962537/

[5962538] rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} tasks/scrub_test}
time:   00:10:31
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962538/

[5962539] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast msgr-failures/osd-delay rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-snaps-few-objects-overwrites}
time:   00:26:03
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962539/

[5962540] rados/singleton-nomsgr/{all/lazy_omap_stats_output rados supported-random-distro$/{rhel_latest}}
time:   00:11:30
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962540/

[5962541] rados/cephadm/smoke-roleless/{distro/centos_7 start}
time:   00:14:10
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962541/

[5962543] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-many-deletes}
time:   00:21:05
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962543/

[5962544] rados/singleton/{all/thrash-backfill-full msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}}
time:   00:49:53
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962544/

[5962545] rados/dashboard/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-hybrid supported-random-distro$/{centos_latest} tasks/dashboard}
time:   01:04:28
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962545/

[5962546] rados/objectstore/{backends/fusestore supported-random-distro$/{rhel_latest}}
time:   00:08:50
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962546/

[5962547] rados/singleton-bluestore/{all/cephtool msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}}
time:   00:27:57
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962547/

[5962548] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/off msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/cache-pool-snaps-readproxy}
time:   00:24:38
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962548/

[5962549] rados/cephadm/workunits/{distro/centos_latest task/test_adoption}
time:   00:07:53
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962549/

[5962550] rados/standalone/{supported-random-distro$/{rhel_latest} workloads/mon}
time:   00:47:50
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962550/

[5962551] rados/perf/{ceph objectstore/bluestore-low-osd-mem-target openstack settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read}
time:   00:11:08
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962551/

[5962552] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
time:   00:12:27
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962552/

[5962553] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/packaged msgr/async-v2only start tasks/rados_python}
time:   00:21:43
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962553/

[5962554] rados/singleton/{all/thrash-eio msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_latest}}
time:   00:38:39
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962554/

[5962555] rados/singleton-nomsgr/{all/librados_hello_world rados supported-random-distro$/{rhel_latest}}
time:   00:11:57
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962555/

[5962556] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 start}
time:   00:14:02
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962556/

[5962557] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_latest} thrashers/default thrashosds-health workloads/cache-pool-snaps}
time:   00:27:18
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962557/

[5962558] rados/cephadm/smoke-roleless/{distro/centos_8.0 start}
time:   00:12:53
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962558/

[5962559] rados/multimon/{clusters/6 msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_latest} tasks/mon_clock_with_skews}
time:   00:09:14
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962559/

[5962560] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{rhel_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2}
time:   00:26:08
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962560/

[5962561] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
time:   00:28:58
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962561/

[5962562] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados tasks/mon_recovery validater/lockdep}
time:   00:11:29
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962562/

[5962563] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}}
time:   00:18:25
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962563/

[5962564] rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/off distro$/{ubuntu_18.04} msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/radosbench}
time:   01:42:07
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962564/

[5962565] rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/many msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} tasks/rados_api_tests}
time:   00:22:10
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962565/

[5962566] rados/monthrash/{ceph clusters/3-mons msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_latest} thrashers/sync-many workloads/pool-create-delete}
time:   00:16:28
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962566/

[5962568] rados/mgr/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-comp-snappy supported-random-distro$/{centos_latest} tasks/module_selftest}
time:   00:28:02
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962568/

[5962569] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache-snaps-balanced}
time:   00:19:43
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962569/

[5962570] rados/perf/{ceph objectstore/bluestore-stupid openstack settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read}
time:   00:09:32
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962570/

[5962573] rados/cephadm/smoke/{distro/centos_latest fixed-2 start}
time:   00:12:11
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962573/

[5962574] rados/singleton-nomsgr/{all/msgr rados supported-random-distro$/{ubuntu_latest}}
time:   00:15:34
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962574/

[5962575] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-small-objects}
time:   00:20:22
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962575/

[5962576] rados/cephadm/smoke-roleless/{distro/centos_latest start}
time:   00:11:03
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962576/

[5962577] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/off msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-snaps}
time:   00:20:26
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962577/

[5962578] rados/singleton/{all/watch-notify-same-primary msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}}
time:   00:07:39
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962578/

[5962579] rados/dashboard/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-low-osd-mem-target supported-random-distro$/{rhel_latest} tasks/dashboard}
time:   01:06:46
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962579/

[5962580] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{ubuntu_latest}}
time:   00:05:57
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962580/

[5962581] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal msgr-failures/fastclose rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-snaps-few-objects-overwrites}
time:   00:26:36
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962581/

[5962582] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
time:   00:13:53
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962582/

[5962583] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest}
time:   00:19:11
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962583/

[5962584] rados/cephadm/workunits/{distro/centos_latest task/test_cephadm_repos}
time:   00:04:55
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962584/

[5962585] rados/perf/{ceph objectstore/bluestore-basic-min-osd-mem-target openstack settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read}
time:   00:10:01
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962585/

[5962587] rados/singleton/{all/admin-socket msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_latest}}
time:   00:09:01
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962587/

[5962588] rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/rados_cls_all}
time:   00:12:05
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962588/

[5962589] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap msgr-failures/fastclose msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache}
time:   00:08:19
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962589/

[5962591] rados/multimon/{clusters/9 msgr-failures/few msgr/async no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_latest} tasks/mon_recovery}
time:   00:15:44
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962591/

[5962592] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}
time:   00:12:28
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962592/

[5962593] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
time:   00:26:28
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962593/

[5962594] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind}
time:   00:39:31
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962594/

[5962595] rados/singleton-nomsgr/{all/multi-backfill-reject rados supported-random-distro$/{ubuntu_latest}}
time:   00:19:37
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962595/

[5962596] rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/rbd_cls}
time:   00:16:37
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962596/

[5962597] rados/monthrash/{ceph clusters/9-mons msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_5925}
time:   00:08:34
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962597/

[5962598] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/packaged msgr/async start tasks/rados_python}
time:   00:22:12
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962598/

[5962600] rados/standalone/{supported-random-distro$/{rhel_latest} workloads/osd}
time:   02:53:33
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962600/

[5962601] rados/singleton/{all/deduptool msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}}
time:   00:06:40
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962601/

[5962603] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_latest} thrashers/pggrow thrashosds-health workloads/dedup_tier}
time:   00:12:44
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962603/

[5962604] rados/perf/{ceph objectstore/bluestore-bitmap openstack settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read}
time:   00:09:01
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962604/

[5962605] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 start}
time:   00:15:24
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962605/

[5962606] rados/singleton/{all/divergent_priors msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}}
time:   00:08:09
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962606/

[5962607] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
time:   00:26:15
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962607/

[5962608] rados/cephadm/smoke-roleless/{distro/rhel_8.0 start}
time:   00:14:28
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962608/

[5962609] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
time:   00:13:47
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962609/

[5962610] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/off msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/pool-snaps-few-objects}
time:   00:29:27
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962610/

[5962611] rados/singleton-nomsgr/{all/osd_stale_reads rados supported-random-distro$/{centos_latest}}
time:   00:18:03
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962611/

[5962612] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/root msgr/async-v1only start tasks/rados_api_tests}
time:   00:31:29
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962612/

[5962613] rados/singleton/{all/divergent_priors2 msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}}
time:   00:09:13
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962613/

[5962614] rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/many msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/rados_python}
time:   00:12:25
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962614/

[5962615] rados/dashboard/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} tasks/dashboard}
time:   01:05:15
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962615/

[5962616] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{centos_latest}}
time:   00:22:37
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962616/

[5962617] rados/singleton-bluestore/{all/cephtool msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}}
time:   00:27:50
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962617/

[5962619] rados/perf/{ceph objectstore/bluestore-comp openstack settings/optimized ubuntu_latest workloads/radosbench_4M_write}
time:   00:09:18
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962619/

[5962621] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rados_api_tests}
time:   00:20:07
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962621/

[5962622] rados/singleton/{all/dump-stuck msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}}
time:   00:07:37
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962622/

[5962623] rados/multimon/{clusters/21 msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{rhel_latest} tasks/mon_clock_no_skews}
time:   00:09:41
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962623/

[5962624] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2}
time:   00:26:39
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962624/

[5962625] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
time:   00:24:41
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962625/

[5962626] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} msgr-failures/few msgr/async objectstore/bluestore-stupid rados tasks/rados_cls_all validater/lockdep}
time:   00:13:24
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962626/

[5962627] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast msgr-failures/few rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites}
time:   00:26:35
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962627/

[5962628] rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/off distro$/{ubuntu_18.04} msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/snaps-few-objects}
time:   00:35:03
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962628/

[5962629] rados/cephadm/smoke/{distro/rhel_latest fixed-2 start}
time:   00:13:21
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962629/

[5962630] rados/mgr/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} tasks/prometheus}
time:   00:09:40
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962630/

[5962631] rados/monthrash/{ceph clusters/3-mons msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests}
time:   00:19:53
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962631/

[5962632] rados/cephadm/smoke-roleless/{distro/rhel_latest start}
time:   00:12:48
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962632/

[5962633] rados/singleton-nomsgr/{all/pool-access rados supported-random-distro$/{ubuntu_latest}}
time:   00:06:32
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962633/

[5962634] rados/singleton/{all/ec-lost-unfound msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_latest}}
time:   00:31:52
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962634/

[5962635] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_latest} thrashers/mapgap thrashosds-health workloads/radosbench-high-concurrency}
time:   00:45:36
info:   http://pulpito.ceph.com/yuriw-2021-03-13_21:47:56-rados-wip-yuriw-octopus-15.2.10-distro-basic-smithi/5962635/