2020-12-28T07:27:17.366 INFO:root:teuthology version: 1.0.0-8e49dda4 2020-12-28T07:27:17.367 INFO:teuthology.results:subset: '1/100000' 2020-12-28T07:27:17.368 INFO:teuthology.results:seed: '3567' 2020-12-28T12:14:09.653 INFO:root:teuthology version: 1.0.0-8e49dda4 2020-12-28T12:14:09.687 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-12-29T00:04:35.488 INFO:teuthology.results:Tests finished! gathering results... 2020-12-29T00:05:29.094 INFO:scrape:Found 367 jobs 2020-12-29T00:05:29.480 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8592-g3ef56c39 (3ef56c398ba628e9011753b67786b8e2a37447a6) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f1f1b125a2b] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x56176cc39d60] 2020-12-29T00:05:29.483 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-12-28T12:08:49.684 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-12-28T12:08:50.022 INFO:teuthology.orchestra.run.smithi179.stdout: 2020-12-29T00:05:29.484 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-28T12:10:12.020 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-28T12:10:12.103 INFO:tasks.ceph.osd.4:Stopped , 2020-12-28T12:10:12.103 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2020-12-29T00:05:29.486 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-28T12:12:01.785 INFO:teuthology.orchestra.run.smithi179.stdout: remote refid st t when poll reach delay offset jitter , 2020-12-28T12:12:01.786 INFO:teuthology.orchestra.run.smithi179.stdout:============================================================================== , 2020-12-28T12:12:01.786 INFO:teuthology.orchestra.run.smithi179.stdout:+hv01.front.sepi 216.240.36.24 3 u 7 64 377 0.069 -7.145 7.272 2020-12-29T00:05:29.582 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8592-g3ef56c39 (3ef56c398ba628e9011753b67786b8e2a37447a6) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f1f1b125a2b] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x56176cc39d60] 2020-12-29T00:05:29.584 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-12-28T12:08:49.684 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-12-28T12:08:50.022 INFO:teuthology.orchestra.run.smithi179.stdout: 2020-12-29T00:05:29.585 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-28T12:10:12.020 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-28T12:10:12.103 INFO:tasks.ceph.osd.4:Stopped , 2020-12-28T12:10:12.103 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2020-12-29T00:05:29.585 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-28T12:12:01.785 INFO:teuthology.orchestra.run.smithi179.stdout: remote refid st t when poll reach delay offset jitter , 2020-12-28T12:12:01.786 INFO:teuthology.orchestra.run.smithi179.stdout:============================================================================== , 2020-12-28T12:12:01.786 INFO:teuthology.orchestra.run.smithi179.stdout:+hv01.front.sepi 216.240.36.24 3 u 7 64 377 0.069 -7.145 7.272 2020-12-29T00:05:29.683 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8592-g3ef56c39 (3ef56c398ba628e9011753b67786b8e2a37447a6) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f1f1b125a2b] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x56176cc39d60] 2020-12-29T00:05:29.685 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-12-28T12:08:49.684 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2020-12-28T12:08:50.022 INFO:teuthology.orchestra.run.smithi179.stdout: 2020-12-29T00:05:29.686 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-28T12:10:12.020 INFO:teuthology.orchestra.run:waiting for 300 , 2020-12-28T12:10:12.103 INFO:tasks.ceph.osd.4:Stopped , 2020-12-28T12:10:12.103 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2020-12-29T00:05:29.687 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-28T12:12:01.785 INFO:teuthology.orchestra.run.smithi179.stdout: remote refid st t when poll reach delay offset jitter , 2020-12-28T12:12:01.786 INFO:teuthology.orchestra.run.smithi179.stdout:============================================================================== , 2020-12-28T12:12:01.786 INFO:teuthology.orchestra.run.smithi179.stdout:+hv01.front.sepi 216.240.36.24 3 u 7 64 377 0.069 -7.145 7.272 2020-12-29T00:05:30.297 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-8592-g3ef56c39 (3ef56c398ba628e9011753b67786b8e2a37447a6) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f8c99ebda2b] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x59c) [0x565213056c4c] 2020-12-29T00:05:30.298 WARNING:scrape:Ignoring malparsed backtrace: 1 , 2 , abort 2020-12-29T00:05:30.337 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-28T10:29:39.158 INFO:teuthology.orchestra.run.smithi092.stdout:64424509477 , 2020-12-28T10:29:39.173 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509477 got 64424509477 for osd.1 , 2020-12-28T10:29:39.173 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2020-12-29T00:05:30.343 WARNING:scrape:Ignoring malparsed backtrace: dumped all , 2020-12-28T10:30:03.114 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2020-12-28T10:30:02.014517+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22591064938,"num_objects":7584,"num_object_clones":0,"num_object_copies":26392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5597,"num_whiteouts":3,"num_read":11198,"num_read_kb":5604,"num_write":30588,"num_write_kb":18042095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":830,"num_bytes_recovered":2318619681,"num_keys_recovered":2,"num_objects_omap":3,"num_objects_hit_set_archive":21,"num_bytes_hit_set_archive":12303,"num_flush":3643,"num_flush_kb":10623124,"num_evict":1682,"num_evict_kb":4889582,"num_promote":5602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15669,"ondisk_log_size":15669,"up":115,"acting":113,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":115,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":31045320,"kb_used_data":30614768,"kb_used_omap":0,"kb_used_meta":419583,"kb_avail":723929400,"statfs":{"total":773094113280,"available":741303705600,"internally_reserved":0,"allocated":31349522432,"data_stored":37113846502,"data_compressed":602711681,"data_compressed_allocated":5885046784,"data_compressed_original":11770093568,"omap_allocated":964,"internal_metadata":429653052},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-355,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-363,"num_bytes_recovered":-1229979446,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002045"},"pg_stats":[{"pgid":"2.11","version":"0'0","reported_seq":"76","reported_epoch":"117","state":"active+clean","last_fresh":"2020-12-28T10:29:57.117116+0000","last_change":"2020-12-28T10:29:18.165947+0000","last_active":"2020-12-28T10:29:57.117116+0000","last_peered":"2020-12-28T10:29:57.117116+0000","last_clean":"2020-12-28T10:29:57.117116+0000","last_became_active":"2020-12-28T10:29:18.165439+0000","last_became_peered":"2020-12-28T10:29:18.165439+0000","last_unstale":"2020-12-28T10:29:57.117116+0000","last_undegraded":"2020-12-28T10:29:57.117116+0000","last_fullsized":"2020-12-28T10:29:57.117116+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:28:00.453819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:28:00.453819+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"92'7542","reported_seq":"10152","reported_epoch":"118","state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2020-12-28T10:29:58.481266+0000","last_change":"2020-12-28T10:29:56.272678+0000","last_active":"2020-12-28T10:29:58.481266+0000","last_peered":"2020-12-28T10:29:58.481266+0000","last_clean":"2020-12-28T10:28:10.052115+0000","last_became_active":"2020-12-28T10:29:19.388633+0000","last_became_peered":"2020-12-28T10:29:19.388633+0000","last_unstale":"2020-12-28T10:29:58.481266+0000","last_undegraded":"2020-12-28T10:29:18.926010+0000","last_fullsized":"2020-12-28T10:29:18.911586+0000","mapping_epoch":95,"log_start":"75'4541","ondisk_log_start":"75'4541","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:27:10.167150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:27:10.167150+0000","last_clean_scrub_stamp":"2020-12-28T10:27:10.167150+0000","log_size":3001,"ondisk_log_size":3001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2955629649,"num_objects":984,"num_object_clones":0,"num_object_copies":2952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":886,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":489,"num_whiteouts":0,"num_read":2842,"num_read_kb":1421,"num_write":6866,"num_write_kb":1887177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":93724420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":4,"num_bytes_hit_set_archive":1946,"num_flush":933,"num_flush_kb":2735053,"num_evict":443,"num_evict_kb":1287494,"num_promote":1423,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":984}],"blocked_by":[],"up_primary":4,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"120","reported_epoch":"117","state":"active+clean","last_fresh":"2020-12-28T10:29:57.118302+0000","last_change":"2020-12-28T10:28:25.647913+0000","last_active":"2020-12-28T10:29:57.118302+0000","last_peered":"2020-12-28T10:29:57.118302+0000","last_clean":"2020-12-28T10:29:57.118302+0000","last_became_active":"2020-12-28T10:28:07.912797+0000","last_became_peered":"2020-12-28T10:28:07.912797+0000","last_unstale":"2020-12-28T10:29:57.118302+0000","last_undegraded":"2020-12-28T10:29:57.118302+0000","last_fullsized":"2020-12-28T10:29:57.118302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:28:25.647771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:28:25.647771+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"17'2","reported_seq":"85","reported_epoch":"117","state":"active+clean","last_fresh":"2020-12-28T10:29:57.099548+0000","last_change":"2020-12-28T10:28:12.706409+0000","last_active":"2020-12-28T10:29:57.099548+0000","last_peered":"2020-12-28T10:29:57.099548+0000","last_clean":"2020-12-28T10:29:57.099548+0000","last_became_active":"2020-12-28T10:28:06.931370+0000","last_became_peered":"2020-12-28T10:28:06.931370+0000","last_unstale":"2020-12-28T10:29:57.099548+0000","last_undegraded":"2020-12-28T10:29:57.099548+0000","last_fullsized":"2020-12-28T10:29:57.099548+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":3,"last_scrub":"17'2","last_scrub_stamp":"2020-12-28T10:28:12.706274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:44.957484+0000","last_clean_scrub_stamp":"2020-12-28T10:28:12.706274+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"90'7269","reported_seq":"9692","reported_epoch":"117","state":"active+clean","last_fresh":"2020-12-28T10:29:57.099127+0000","last_change":"2020-12-28T10:29:05.014780+0000","last_active":"2020-12-28T10:29:57.099127+0000","last_peered":"2020-12-28T10:29:57.099127+0000","last_clean":"2020-12-28T10:29:57.099127+0000","last_became_active":"2020-12-28T10:27:11.629397+0000","last_became_peered":"2020-12-28T10:27:11.629397+0000","last_unstale":"2020-12-28T10:29:57.099127+0000","last_undegraded":"2020-12-28T10:29:57.099127+0000","last_fullsized":"2020-12-28T10:29:57.099127+0000","mapping_epoch":26,"log_start":"74'4268","ondisk_log_start":"74'4268","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'7202","last_scrub_stamp":"2020-12-28T10:29:05.014618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:27:10.167150+0000","last_clean_scrub_stamp":"2020-12-28T10:29:05.014618+0000","log_size":3001,"ondisk_log_size":3001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2905734603,"num_objects":986,"num_object_clones":0,"num_object_copies":2958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":2803,"num_read_kb":1402,"num_write":6741,"num_write_kb":1851740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":3558,"num_flush":913,"num_flush_kb":2644122,"num_evict":421,"num_evict_kb":1233889,"num_promote":1401,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"130","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:54.207517+0000","last_change":"2020-12-28T10:29:18.168739+0000","last_active":"2020-12-28T10:29:54.207517+0000","last_peered":"2020-12-28T10:29:54.207517+0000","last_clean":"2020-12-28T10:29:54.207517+0000","last_became_active":"2020-12-28T10:29:18.168127+0000","last_became_peered":"2020-12-28T10:29:18.168127+0000","last_unstale":"2020-12-28T10:29:54.207517+0000","last_undegraded":"2020-12-28T10:29:54.207517+0000","last_fullsized":"2020-12-28T10:29:54.207517+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:26:53.017342+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"17'2","reported_seq":"52","reported_epoch":"118","state":"active+clean","last_fresh":"2020-12-28T10:29:58.110371+0000","last_change":"2020-12-28T10:29:31.932411+0000","last_active":"2020-12-28T10:29:58.110371+0000","last_peered":"2020-12-28T10:29:58.110371+0000","last_clean":"2020-12-28T10:29:58.110371+0000","last_became_active":"2020-12-28T10:29:18.168215+0000","last_became_peered":"2020-12-28T10:29:18.168215+0000","last_unstale":"2020-12-28T10:29:58.110371+0000","last_undegraded":"2020-12-28T10:29:58.110371+0000","last_fullsized":"2020-12-28T10:29:58.110371+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:26:44.957484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:44.957484+0000","last_clean_scrub_stamp":"2020-12-28T10:26:44.957484+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"131","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:54.211703+0000","last_change":"2020-12-28T10:29:18.175803+0000","last_active":"2020-12-28T10:29:54.211703+0000","last_peered":"2020-12-28T10:29:54.211703+0000","last_clean":"2020-12-28T10:29:54.211703+0000","last_became_active":"2020-12-28T10:29:18.174884+0000","last_became_peered":"2020-12-28T10:29:18.174884+0000","last_unstale":"2020-12-28T10:29:54.211703+0000","last_undegraded":"2020-12-28T10:29:54.211703+0000","last_fullsized":"2020-12-28T10:29:54.211703+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:26:53.017342+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"90'7052","reported_seq":"9397","reported_epoch":"117","state":"active+clean","last_fresh":"2020-12-28T10:29:57.129776+0000","last_change":"2020-12-28T10:28:43.058209+0000","last_active":"2020-12-28T10:29:57.129776+0000","last_peered":"2020-12-28T10:29:57.129776+0000","last_clean":"2020-12-28T10:29:57.129776+0000","last_became_active":"2020-12-28T10:27:11.629316+0000","last_became_peered":"2020-12-28T10:27:11.629316+0000","last_unstale":"2020-12-28T10:29:57.129776+0000","last_undegraded":"2020-12-28T10:29:57.129776+0000","last_fullsized":"2020-12-28T10:29:57.129776+0000","mapping_epoch":26,"log_start":"72'4051","ondisk_log_start":"72'4051","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"81'5567","last_scrub_stamp":"2020-12-28T10:28:43.058063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:27:10.167150+0000","last_clean_scrub_stamp":"2020-12-28T10:28:43.058063+0000","log_size":3001,"ondisk_log_size":3001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2935489361,"num_objects":986,"num_object_clones":0,"num_object_copies":2958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":1,"num_read":2722,"num_read_kb":1362,"num_write":6536,"num_write_kb":1803479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":6,"num_bytes_hit_set_archive":3472,"num_flush":872,"num_flush_kb":2554180,"num_evict":382,"num_evict_kb":1123715,"num_promote":1362,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"17'2","reported_seq":"91","reported_epoch":"117","state":"active+clean","last_fresh":"2020-12-28T10:29:57.118154+0000","last_change":"2020-12-28T10:28:13.720167+0000","last_active":"2020-12-28T10:29:57.118154+0000","last_peered":"2020-12-28T10:29:57.118154+0000","last_clean":"2020-12-28T10:29:57.118154+0000","last_became_active":"2020-12-28T10:28:06.926449+0000","last_became_peered":"2020-12-28T10:28:06.926449+0000","last_unstale":"2020-12-28T10:29:57.118154+0000","last_undegraded":"2020-12-28T10:29:57.118154+0000","last_fullsized":"2020-12-28T10:29:57.118154+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":3,"last_scrub":"17'2","last_scrub_stamp":"2020-12-28T10:28:13.720041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:44.957484+0000","last_clean_scrub_stamp":"2020-12-28T10:28:13.720041+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"92'933","reported_seq":"1022","reported_epoch":"117","state":"active+clean","last_fresh":"2020-12-28T10:29:57.099735+0000","last_change":"2020-12-28T10:28:54.178003+0000","last_active":"2020-12-28T10:29:57.099735+0000","last_peered":"2020-12-28T10:29:57.099735+0000","last_clean":"2020-12-28T10:29:57.099735+0000","last_became_active":"2020-12-28T10:27:40.224828+0000","last_became_peered":"2020-12-28T10:27:40.224828+0000","last_unstale":"2020-12-28T10:29:57.099735+0000","last_undegraded":"2020-12-28T10:29:57.099735+0000","last_fullsized":"2020-12-28T10:29:57.099735+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'626","last_scrub_stamp":"2020-12-28T10:28:54.177869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:27:07.133282+0000","last_clean_scrub_stamp":"2020-12-28T10:28:54.177869+0000","log_size":933,"ondisk_log_size":933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2796541282,"num_objects":932,"num_object_clones":0,"num_object_copies":3728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":933,"num_write_kb":2735053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0,2],"acting":[3,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"120","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:54.211175+0000","last_change":"2020-12-28T10:28:17.558424+0000","last_active":"2020-12-28T10:29:54.211175+0000","last_peered":"2020-12-28T10:29:54.211175+0000","last_clean":"2020-12-28T10:29:54.211175+0000","last_became_active":"2020-12-28T10:28:07.941979+0000","last_became_peered":"2020-12-28T10:28:07.941979+0000","last_unstale":"2020-12-28T10:29:54.211175+0000","last_undegraded":"2020-12-28T10:29:54.211175+0000","last_fullsized":"2020-12-28T10:29:54.211175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:28:17.558340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:28:17.558340+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"17'2","reported_seq":"89","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:54.207524+0000","last_change":"2020-12-28T10:29:18.170020+0000","last_active":"2020-12-28T10:29:54.207524+0000","last_peered":"2020-12-28T10:29:54.207524+0000","last_clean":"2020-12-28T10:29:54.207524+0000","last_became_active":"2020-12-28T10:29:18.168271+0000","last_became_peered":"2020-12-28T10:29:18.168271+0000","last_unstale":"2020-12-28T10:29:54.207524+0000","last_undegraded":"2020-12-28T10:29:54.207524+0000","last_fullsized":"2020-12-28T10:29:54.207524+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:26:44.957484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:44.957484+0000","last_clean_scrub_stamp":"2020-12-28T10:26:44.957484+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":1,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"91'925","reported_seq":"2541","reported_epoch":"118","state":"peering","last_fresh":"2020-12-28T10:29:58.106604+0000","last_change":"2020-12-28T10:29:58.107212+0000","last_active":"2020-12-28T10:29:58.106604+0000","last_peered":"2020-12-28T10:29:57.457643+0000","last_clean":"2020-12-28T10:28:10.039532+0000","last_became_active":"2020-12-28T10:29:19.399774+0000","last_became_peered":"2020-12-28T10:29:19.399774+0000","last_unstale":"2020-12-28T10:29:58.106604+0000","last_undegraded":"2020-12-28T10:29:58.106604+0000","last_fullsized":"2020-12-28T10:29:58.106604+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:27:07.133282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:27:07.133282+0000","last_clean_scrub_stamp":"2020-12-28T10:27:07.133282+0000","log_size":925,"ondisk_log_size":925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2749201154,"num_objects":923,"num_object_clones":0,"num_object_copies":3692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":925,"num_write_kb":2689769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":742,"num_bytes_recovered":2201625494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6,4],"acting":[3,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5,6],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'2","reported_seq":"116","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:55.535631+0000","last_change":"2020-12-28T10:29:55.535631+0000","last_active":"2020-12-28T10:29:55.535631+0000","last_peered":"2020-12-28T10:29:55.535631+0000","last_clean":"2020-12-28T10:29:55.535631+0000","last_became_active":"2020-12-28T10:28:02.003741+0000","last_became_peered":"2020-12-28T10:28:02.003741+0000","last_unstale":"2020-12-28T10:29:55.535631+0000","last_undegraded":"2020-12-28T10:29:55.535631+0000","last_fullsized":"2020-12-28T10:29:55.535631+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'2","last_scrub_stamp":"2020-12-28T10:28:02.525871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:44.957484+0000","last_clean_scrub_stamp":"2020-12-28T10:28:02.525871+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"121","reported_epoch":"117","state":"active+clean","last_fresh":"2020-12-28T10:29:57.130515+0000","last_change":"2020-12-28T10:28:38.757039+0000","last_active":"2020-12-28T10:29:57.130515+0000","last_peered":"2020-12-28T10:29:57.130515+0000","last_clean":"2020-12-28T10:29:57.130515+0000","last_became_active":"2020-12-28T10:28:07.929443+0000","last_became_peered":"2020-12-28T10:28:07.929443+0000","last_unstale":"2020-12-28T10:29:57.130515+0000","last_undegraded":"2020-12-28T10:29:57.130515+0000","last_fullsized":"2020-12-28T10:29:57.130515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:28:38.756489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:28:38.756489+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"90'913","reported_seq":"1033","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:54.207304+0000","last_change":"2020-12-28T10:28:55.869392+0000","last_active":"2020-12-28T10:29:54.207304+0000","last_peered":"2020-12-28T10:29:54.207304+0000","last_clean":"2020-12-28T10:29:54.207304+0000","last_became_active":"2020-12-28T10:27:10.283559+0000","last_became_peered":"2020-12-28T10:27:10.283559+0000","last_unstale":"2020-12-28T10:29:54.207304+0000","last_undegraded":"2020-12-28T10:29:54.207304+0000","last_fullsized":"2020-12-28T10:29:54.207304+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'735","last_scrub_stamp":"2020-12-28T10:28:55.869260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:27:07.133282+0000","last_clean_scrub_stamp":"2020-12-28T10:28:55.869260+0000","log_size":913,"ondisk_log_size":913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2707114918,"num_objects":913,"num_object_clones":0,"num_object_copies":3652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":913,"num_write_kb":2644122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,5],"acting":[1,2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"66","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:53.911751+0000","last_change":"2020-12-28T10:29:26.273156+0000","last_active":"2020-12-28T10:29:53.911751+0000","last_peered":"2020-12-28T10:29:53.911751+0000","last_clean":"2020-12-28T10:29:53.911751+0000","last_became_active":"2020-12-28T10:29:26.272353+0000","last_became_peered":"2020-12-28T10:29:26.272353+0000","last_unstale":"2020-12-28T10:29:53.911751+0000","last_undegraded":"2020-12-28T10:29:53.911751+0000","last_fullsized":"2020-12-28T10:29:53.911751+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:28:49.962084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:28:49.962084+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'2","reported_seq":"119","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:54.210812+0000","last_change":"2020-12-28T10:29:18.175552+0000","last_active":"2020-12-28T10:29:54.210812+0000","last_peered":"2020-12-28T10:29:54.210812+0000","last_clean":"2020-12-28T10:29:54.210812+0000","last_became_active":"2020-12-28T10:29:18.174850+0000","last_became_peered":"2020-12-28T10:29:18.174850+0000","last_unstale":"2020-12-28T10:29:54.210812+0000","last_undegraded":"2020-12-28T10:29:54.210812+0000","last_fullsized":"2020-12-28T10:29:54.210812+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:26:44.957484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:44.957484+0000","last_clean_scrub_stamp":"2020-12-28T10:26:44.957484+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":1,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"90'872","reported_seq":"985","reported_epoch":"118","state":"active+clean","last_fresh":"2020-12-28T10:29:58.124032+0000","last_change":"2020-12-28T10:29:58.122528+0000","last_active":"2020-12-28T10:29:58.124032+0000","last_peered":"2020-12-28T10:29:58.124032+0000","last_clean":"2020-12-28T10:29:58.124032+0000","last_became_active":"2020-12-28T10:29:58.121167+0000","last_became_peered":"2020-12-28T10:29:58.121167+0000","last_unstale":"2020-12-28T10:29:58.124032+0000","last_undegraded":"2020-12-28T10:29:58.124032+0000","last_fullsized":"2020-12-28T10:29:58.124032+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:27:07.133282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:27:07.133282+0000","last_clean_scrub_stamp":"2020-12-28T10:27:07.133282+0000","log_size":872,"ondisk_log_size":872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2615028768,"num_objects":872,"num_object_clones":0,"num_object_copies":3488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":872,"num_write_kb":2554180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,6],"acting":[4,5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"125","reported_epoch":"117","state":"active+clean","last_fresh":"2020-12-28T10:29:57.131705+0000","last_change":"2020-12-28T10:28:49.690414+0000","last_active":"2020-12-28T10:29:57.131705+0000","last_peered":"2020-12-28T10:29:57.131705+0000","last_clean":"2020-12-28T10:29:57.131705+0000","last_became_active":"2020-12-28T10:28:07.936804+0000","last_became_peered":"2020-12-28T10:28:07.936804+0000","last_unstale":"2020-12-28T10:29:57.131705+0000","last_undegraded":"2020-12-28T10:29:57.131705+0000","last_fullsized":"2020-12-28T10:29:57.131705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:28:49.690271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:28:49.690271+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"17'2","reported_seq":"90","reported_epoch":"117","state":"active+clean","last_fresh":"2020-12-28T10:29:57.130391+0000","last_change":"2020-12-28T10:28:37.712031+0000","last_active":"2020-12-28T10:29:57.130391+0000","last_peered":"2020-12-28T10:29:57.130391+0000","last_clean":"2020-12-28T10:29:57.130391+0000","last_became_active":"2020-12-28T10:28:06.970929+0000","last_became_peered":"2020-12-28T10:28:06.970929+0000","last_unstale":"2020-12-28T10:29:57.130391+0000","last_undegraded":"2020-12-28T10:29:57.130391+0000","last_fullsized":"2020-12-28T10:29:57.130391+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":4,"last_scrub":"17'2","last_scrub_stamp":"2020-12-28T10:28:37.711820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:44.957484+0000","last_clean_scrub_stamp":"2020-12-28T10:28:37.711820+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"91'7447","reported_seq":"9940","reported_epoch":"117","state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2020-12-28T10:29:58.082741+0000","last_change":"2020-12-28T10:29:57.458678+0000","last_active":"2020-12-28T10:29:58.082741+0000","last_peered":"2020-12-28T10:29:58.082741+0000","last_clean":"2020-12-28T10:28:10.150690+0000","last_became_active":"2020-12-28T10:29:19.419249+0000","last_became_peered":"2020-12-28T10:29:19.419249+0000","last_unstale":"2020-12-28T10:29:58.082741+0000","last_undegraded":"2020-12-28T10:29:18.914771+0000","last_fullsized":"2020-12-28T10:29:18.909895+0000","mapping_epoch":95,"log_start":"75'4446","ondisk_log_start":"75'4446","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:27:10.167150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:27:10.167150+0000","last_clean_scrub_stamp":"2020-12-28T10:27:10.167150+0000","log_size":3001,"ondisk_log_size":3001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2926325203,"num_objects":985,"num_object_clones":0,"num_object_copies":2955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":489,"num_whiteouts":2,"num_read":2831,"num_read_kb":1419,"num_write":6801,"num_write_kb":1876574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":23269767,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":5,"num_bytes_hit_set_archive":3327,"num_flush":925,"num_flush_kb":2689769,"num_evict":436,"num_evict_kb":1244484,"num_promote":1416,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":985}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"123","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:54.207482+0000","last_change":"2020-12-28T10:28:12.145089+0000","last_active":"2020-12-28T10:29:54.207482+0000","last_peered":"2020-12-28T10:29:54.207482+0000","last_clean":"2020-12-28T10:29:54.207482+0000","last_became_active":"2020-12-28T10:28:07.941514+0000","last_became_peered":"2020-12-28T10:28:07.941514+0000","last_unstale":"2020-12-28T10:29:54.207482+0000","last_undegraded":"2020-12-28T10:29:54.207482+0000","last_fullsized":"2020-12-28T10:29:54.207482+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:28:12.144923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:28:12.144923+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"17'2","reported_seq":"91","reported_epoch":"117","state":"active+clean","last_fresh":"2020-12-28T10:29:57.118422+0000","last_change":"2020-12-28T10:28:12.741372+0000","last_active":"2020-12-28T10:29:57.118422+0000","last_peered":"2020-12-28T10:29:57.118422+0000","last_clean":"2020-12-28T10:29:57.118422+0000","last_became_active":"2020-12-28T10:28:06.934451+0000","last_became_peered":"2020-12-28T10:28:06.934451+0000","last_unstale":"2020-12-28T10:29:57.118422+0000","last_undegraded":"2020-12-28T10:29:57.118422+0000","last_fullsized":"2020-12-28T10:29:57.118422+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":3,"last_scrub":"17'2","last_scrub_stamp":"2020-12-28T10:28:12.741288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:44.957484+0000","last_clean_scrub_stamp":"2020-12-28T10:28:12.741288+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"17'2","reported_seq":"73","reported_epoch":"117","state":"active+clean","last_fresh":"2020-12-28T10:29:57.130175+0000","last_change":"2020-12-28T10:28:16.738909+0000","last_active":"2020-12-28T10:29:57.130175+0000","last_peered":"2020-12-28T10:29:57.130175+0000","last_clean":"2020-12-28T10:29:57.130175+0000","last_became_active":"2020-12-28T10:28:06.933486+0000","last_became_peered":"2020-12-28T10:28:06.933486+0000","last_unstale":"2020-12-28T10:29:57.130175+0000","last_undegraded":"2020-12-28T10:29:57.130175+0000","last_fullsized":"2020-12-28T10:29:57.130175+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":4,"last_scrub":"17'2","last_scrub_stamp":"2020-12-28T10:28:16.738750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:44.957484+0000","last_clean_scrub_stamp":"2020-12-28T10:28:16.738750+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":"131","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:54.211507+0000","last_change":"2020-12-28T10:29:18.179872+0000","last_active":"2020-12-28T10:29:54.211507+0000","last_peered":"2020-12-28T10:29:54.211507+0000","last_clean":"2020-12-28T10:29:54.211507+0000","last_became_active":"2020-12-28T10:29:18.179291+0000","last_became_peered":"2020-12-28T10:29:18.179291+0000","last_unstale":"2020-12-28T10:29:54.211507+0000","last_undegraded":"2020-12-28T10:29:54.211507+0000","last_fullsized":"2020-12-28T10:29:54.211507+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:26:53.017342+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.9","version":"17'2","reported_seq":"113","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:54.210824+0000","last_change":"2020-12-28T10:28:14.506379+0000","last_active":"2020-12-28T10:29:54.210824+0000","last_peered":"2020-12-28T10:29:54.210824+0000","last_clean":"2020-12-28T10:29:54.210824+0000","last_became_active":"2020-12-28T10:28:06.929521+0000","last_became_peered":"2020-12-28T10:28:06.929521+0000","last_unstale":"2020-12-28T10:29:54.210824+0000","last_undegraded":"2020-12-28T10:29:54.210824+0000","last_fullsized":"2020-12-28T10:29:54.210824+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":4,"last_scrub":"17'2","last_scrub_stamp":"2020-12-28T10:28:14.506230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:44.957484+0000","last_clean_scrub_stamp":"2020-12-28T10:28:14.506230+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":"127","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:54.206797+0000","last_change":"2020-12-28T10:28:38.540211+0000","last_active":"2020-12-28T10:29:54.206797+0000","last_peered":"2020-12-28T10:29:54.206797+0000","last_clean":"2020-12-28T10:29:54.206797+0000","last_became_active":"2020-12-28T10:28:09.918343+0000","last_became_peered":"2020-12-28T10:28:09.918343+0000","last_unstale":"2020-12-28T10:29:54.206797+0000","last_undegraded":"2020-12-28T10:29:54.206797+0000","last_fullsized":"2020-12-28T10:29:54.206797+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:28:38.540092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:28:38.540092+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"17'2","reported_seq":"73","reported_epoch":"117","state":"active+clean","last_fresh":"2020-12-28T10:29:57.131381+0000","last_change":"2020-12-28T10:28:17.736899+0000","last_active":"2020-12-28T10:29:57.131381+0000","last_peered":"2020-12-28T10:29:57.131381+0000","last_clean":"2020-12-28T10:29:57.131381+0000","last_became_active":"2020-12-28T10:28:06.945387+0000","last_became_peered":"2020-12-28T10:28:06.945387+0000","last_unstale":"2020-12-28T10:29:57.131381+0000","last_undegraded":"2020-12-28T10:29:57.131381+0000","last_fullsized":"2020-12-28T10:29:57.131381+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":4,"last_scrub":"17'2","last_scrub_stamp":"2020-12-28T10:28:17.736735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:44.957484+0000","last_clean_scrub_stamp":"2020-12-28T10:28:17.736735+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":"64","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:54.206791+0000","last_change":"2020-12-28T10:28:16.081882+0000","last_active":"2020-12-28T10:29:54.206791+0000","last_peered":"2020-12-28T10:29:54.206791+0000","last_clean":"2020-12-28T10:29:54.206791+0000","last_became_active":"2020-12-28T10:28:09.927743+0000","last_became_peered":"2020-12-28T10:28:09.927743+0000","last_unstale":"2020-12-28T10:29:54.206791+0000","last_undegraded":"2020-12-28T10:29:54.206791+0000","last_fullsized":"2020-12-28T10:29:54.206791+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:28:16.081778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:28:16.081778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":"128","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:54.210958+0000","last_change":"2020-12-28T10:28:39.659681+0000","last_active":"2020-12-28T10:29:54.210958+0000","last_peered":"2020-12-28T10:29:54.210958+0000","last_clean":"2020-12-28T10:29:54.210958+0000","last_became_active":"2020-12-28T10:28:09.929292+0000","last_became_peered":"2020-12-28T10:28:09.929292+0000","last_unstale":"2020-12-28T10:29:54.210958+0000","last_undegraded":"2020-12-28T10:29:54.210958+0000","last_fullsized":"2020-12-28T10:29:54.210958+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:28:39.659522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:28:39.659522+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":"27","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:53.911569+0000","last_change":"2020-12-28T10:29:26.272880+0000","last_active":"2020-12-28T10:29:53.911569+0000","last_peered":"2020-12-28T10:29:53.911569+0000","last_clean":"2020-12-28T10:29:53.911569+0000","last_became_active":"2020-12-28T10:29:26.272244+0000","last_became_peered":"2020-12-28T10:29:26.272244+0000","last_unstale":"2020-12-28T10:29:53.911569+0000","last_undegraded":"2020-12-28T10:29:53.911569+0000","last_fullsized":"2020-12-28T10:29:53.911569+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:28:50.631577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:28:50.631577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":"75","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:54.207512+0000","last_change":"2020-12-28T10:29:18.168818+0000","last_active":"2020-12-28T10:29:54.207512+0000","last_peered":"2020-12-28T10:29:54.207512+0000","last_clean":"2020-12-28T10:29:54.207512+0000","last_became_active":"2020-12-28T10:29:18.168241+0000","last_became_peered":"2020-12-28T10:29:18.168241+0000","last_unstale":"2020-12-28T10:29:54.207512+0000","last_undegraded":"2020-12-28T10:29:54.207512+0000","last_fullsized":"2020-12-28T10:29:54.207512+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:26:53.017342+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":"53","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:56.139746+0000","last_change":"2020-12-28T10:29:56.139746+0000","last_active":"2020-12-28T10:29:56.139746+0000","last_peered":"2020-12-28T10:29:56.139746+0000","last_clean":"2020-12-28T10:29:56.139746+0000","last_became_active":"2020-12-28T10:29:26.604318+0000","last_became_peered":"2020-12-28T10:29:26.604318+0000","last_unstale":"2020-12-28T10:29:56.139746+0000","last_undegraded":"2020-12-28T10:29:56.139746+0000","last_fullsized":"2020-12-28T10:29:56.139746+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:28:50.965787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:28:50.965787+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":"47","reported_epoch":"118","state":"active+clean","last_fresh":"2020-12-28T10:29:58.110706+0000","last_change":"2020-12-28T10:29:31.937831+0000","last_active":"2020-12-28T10:29:58.110706+0000","last_peered":"2020-12-28T10:29:58.110706+0000","last_clean":"2020-12-28T10:29:58.110706+0000","last_became_active":"2020-12-28T10:29:18.174377+0000","last_became_peered":"2020-12-28T10:29:18.174377+0000","last_unstale":"2020-12-28T10:29:58.110706+0000","last_undegraded":"2020-12-28T10:29:58.110706+0000","last_fullsized":"2020-12-28T10:29:58.110706+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:26:53.017342+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":"82","reported_epoch":"116","state":"active+clean","last_fresh":"2020-12-28T10:29:53.911337+0000","last_change":"2020-12-28T10:29:26.600978+0000","last_active":"2020-12-28T10:29:53.911337+0000","last_peered":"2020-12-28T10:29:53.911337+0000","last_clean":"2020-12-28T10:29:53.911337+0000","last_became_active":"2020-12-28T10:29:26.600150+0000","last_became_peered":"2020-12-28T10:29:26.600150+0000","last_unstale":"2020-12-28T10:29:53.911337+0000","last_undegraded":"2020-12-28T10:29:53.911337+0000","last_fullsized":"2020-12-28T10:29:53.911337+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-12-28T10:28:13.121978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-12-28T10:26:53.017342+0000","last_clean_scrub_stamp":"2020-12-28T10:28:13.121978+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":4,"stat_sum":{"num_bytes":11723178816,"num_objects":3941,"num_object_clones":0,"num_object_copies":11823,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1954,"num_whiteouts":3,"num_read":11198,"num_read_kb":5604,"num_write":26944,"num_write_kb":7418970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":116994187,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":21,"num_bytes_hit_set_archive":12303,"num_flush":3643,"num_flush_kb":10623124,"num_evict":1682,"num_evict_kb":4889582,"num_promote":5602,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15498022912,"data_stored":15381144729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12004,"ondisk_log_size":12004,"up":12,"acting":10,"num_store_stats":7},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":10867886122,"num_objects":3640,"num_object_clones":0,"num_object_copies":14560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3643,"num_write_kb":10623124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":742,"num_bytes_recovered":2201625494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15880802304,"data_stored":21765849088,"data_compressed":602711681,"data_compressed_allocated":5885046784,"data_compressed_original":11770093568,"omap_allocated":0,"internal_metadata":0},"log_size":3643,"ondisk_log_size":3643,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":2,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22,"ondisk_log_size":22,"up":33,"acting":33,"num_store_stats":8},{"poolid":2,"num_pg":18,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":54,"acting":54,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509482,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2667100,"kb_used_data":2607572,"kb_used_omap":0,"kb_used_meta":59520,"kb_avail":91704740,"statfs":{"total":96636764160,"available":93905653760,"internally_reserved":0,"allocated":2670153728,"data_stored":2649501886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":60948480},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 28 10:29:52 2020","interfaces":[{"interface":"back","average":{"1min":1.919,"5min":1.9530000000000001,"15min":2.0510000000000002},"min":{"1min":0.73599999999999999,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":6.8380000000000001,"5min":6.9960000000000004,"15min":6.9960000000000004},"last":2.0369999999999999},{"interface":"front","average":{"1min":1.841,"5min":2.105,"15min":2.2370000000000001},"min":{"1min":0.83199999999999996,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":6.3339999999999996,"5min":6.3339999999999996,"15min":6.3339999999999996},"last":2.1349999999999998}]},{"osd":1,"last update":"Mon Dec 28 10:29:52 2020","interfaces":[{"interface":"back","average":{"1min":2.1669999999999998,"5min":2.0800000000000001,"15min":2.1520000000000001},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":6.9279999999999999,"5min":6.9279999999999999,"15min":6.9279999999999999},"last":1.593},{"interface":"front","average":{"1min":2.2589999999999999,"5min":2.0840000000000001,"15min":2.1339999999999999},"min":{"1min":1.1220000000000001,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":5.4119999999999999,"5min":5.4119999999999999,"15min":5.4119999999999999},"last":1.6200000000000001}]},{"osd":2,"last update":"Mon Dec 28 10:29:52 2020","interfaces":[{"interface":"back","average":{"1min":2.2360000000000002,"5min":41.134,"15min":58.509999999999998},"min":{"1min":0.97699999999999998,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":6.4710000000000001,"5min":980.25199999999995,"15min":980.25199999999995},"last":2.1970000000000001},{"interface":"front","average":{"1min":2.4319999999999999,"5min":2.2490000000000001,"15min":2.3170000000000002},"min":{"1min":1.004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":6.5549999999999997,"5min":6.6230000000000002,"15min":6.6230000000000002},"last":2.266}]},{"osd":3,"last update":"Mon Dec 28 10:29:52 2020","interfaces":[{"interface":"back","average":{"1min":2.4039999999999999,"5min":11.791,"15min":5.5289999999999999},"min":{"1min":0.88600000000000001,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":7.0049999999999999,"5min":814.26099999999997,"15min":814.26099999999997},"last":1.337},{"interface":"front","average":{"1min":2.5470000000000002,"5min":2.395,"15min":2.403},"min":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"max":{"1min":7.2409999999999997,"5min":7.2409999999999997,"15min":7.2409999999999997},"last":1.736}]},{"osd":4,"last update":"Mon Dec 28 10:27:48 2020","interfaces":[{"interface":"back","average":{"1min":2.875,"5min":2.875,"15min":2.875},"min":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"max":{"1min":6.6619999999999999,"5min":6.6619999999999999,"15min":6.6619999999999999},"last":1.6890000000000001},{"interface":"front","average":{"1min":2.6819999999999999,"5min":2.6819999999999999,"15min":2.6819999999999999},"min":{"1min":1.216,"5min":1.216,"15min":1.216},"max":{"1min":3.5670000000000002,"5min":3.5670000000000002,"15min":3.5670000000000002},"last":1.0069999999999999}]},{"osd":5,"last update":"Mon Dec 28 10:29:52 2020","interfaces":[{"interface":"back","average":{"1min":13.756,"5min":21.114000000000001,"15min":26.917999999999999},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"max":{"1min":173.16499999999999,"5min":410.97399999999999,"15min":410.97399999999999},"last":1.7929999999999999},{"interface":"front","average":{"1min":13.831,"5min":16.829999999999998,"15min":20.641999999999999},"min":{"1min":1.141,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":175.053,"5min":292.35399999999998,"15min":292.35399999999998},"last":1.359}]},{"osd":6,"last update":"Mon Dec 28 10:29:52 2020","interfaces":[{"interface":"back","average":{"1min":2.742,"5min":2.98,"15min":3.25},"min":{"1min":1.044,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":6.7690000000000001,"5min":11.333,"15min":11.333},"last":1.4330000000000001},{"interface":"front","average":{"1min":43.289000000000001,"5min":10.798,"15min":5.6079999999999997},"min":{"1min":1.244,"5min":1.1040000000000001,"15min":1.1040000000000001},"max":{"1min":768.08699999999999,"5min":768.08699999999999,"15min":768.08699999999999},"last":1.3939999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509481,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4253836,"kb_used_data":4198276,"kb_used_omap":0,"kb_used_meta":55552,"kb_avail":90118004,"statfs":{"total":96636764160,"available":92280836096,"internally_reserved":0,"allocated":4299034624,"data_stored":5377428259,"data_compressed":111428798,"data_compressed_allocated":1088954368,"data_compressed_original":2177908736,"omap_allocated":0,"internal_metadata":56885248},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 28 10:29:56 2020","interfaces":[{"interface":"back","average":{"1min":1.859,"5min":42.652999999999999,"15min":51.728999999999999},"min":{"1min":0.91100000000000003,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":5.5620000000000003,"5min":981.12800000000004,"15min":981.12800000000004},"last":2.5369999999999999},{"interface":"front","average":{"1min":36,"5min":8.7400000000000002,"15min":4.2329999999999997},"min":{"1min":1.228,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":612.89800000000002,"5min":612.89800000000002,"15min":612.89800000000002},"last":612.89800000000002}]},{"osd":1,"last update":"Mon Dec 28 10:29:56 2020","interfaces":[{"interface":"back","average":{"1min":2.298,"5min":1.994,"15min":1.8600000000000001},"min":{"1min":0.84199999999999997,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":6.657,"5min":8.032,"15min":8.032},"last":1.7649999999999999},{"interface":"front","average":{"1min":46.810000000000002,"5min":43.213000000000001,"15min":41.155000000000001},"min":{"1min":0.90500000000000003,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":809.19899999999996,"5min":849.45399999999995,"15min":849.45399999999995},"last":2.851}]},{"osd":2,"last update":"Mon Dec 28 10:29:56 2020","interfaces":[{"interface":"back","average":{"1min":2.3450000000000002,"5min":2.254,"15min":2.306},"min":{"1min":0.93999999999999995,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":3.9260000000000002,"5min":5.7409999999999997,"15min":5.7409999999999997},"last":2.988},{"interface":"front","average":{"1min":2.0870000000000002,"5min":2.1699999999999999,"15min":2.2799999999999998},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":4.2229999999999999,"5min":5.5300000000000002,"15min":5.5300000000000002},"last":2.403}]},{"osd":3,"last update":"Mon Dec 28 10:29:56 2020","interfaces":[{"interface":"back","average":{"1min":45.154000000000003,"5min":13.334,"15min":8.5139999999999993},"min":{"1min":1.6830000000000001,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":770.32899999999995,"5min":770.32899999999995,"15min":770.32899999999995},"last":3.0699999999999998},{"interface":"front","average":{"1min":2.7829999999999999,"5min":4.2320000000000002,"15min":4.9409999999999998},"min":{"1min":1.571,"5min":0.98999999999999999,"15min":0.98999999999999999},"max":{"1min":6.7859999999999996,"5min":61.508000000000003,"15min":61.508000000000003},"last":2.4609999999999999}]},{"osd":4,"last update":"Mon Dec 28 10:27:50 2020","interfaces":[{"interface":"back","average":{"1min":2.4359999999999999,"5min":2.4359999999999999,"15min":2.4359999999999999},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":3.6629999999999998,"5min":3.6629999999999998,"15min":3.6629999999999998},"last":2.2919999999999998},{"interface":"front","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":8.2240000000000002,"5min":8.2240000000000002,"15min":8.2240000000000002},"last":1.8420000000000001}]},{"osd":5,"last update":"Mon Dec 28 10:29:56 2020","interfaces":[{"interface":"back","average":{"1min":12.115,"5min":4.5999999999999996,"15min":3.3410000000000002},"min":{"1min":1.1180000000000001,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":175.49700000000001,"5min":175.49700000000001,"15min":175.49700000000001},"last":2.6080000000000001},{"interface":"front","average":{"1min":11.923,"5min":4.5570000000000004,"15min":3.2669999999999999},"min":{"1min":1.0940000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"max":{"1min":176.26400000000001,"5min":176.26400000000001,"15min":176.26400000000001},"last":3.1469999999999998}]},{"osd":7,"last update":"Mon Dec 28 10:29:56 2020","interfaces":[{"interface":"back","average":{"1min":2.585,"5min":18.681000000000001,"15min":8.2390000000000008},"min":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"max":{"1min":6.4009999999999998,"5min":856.17700000000002,"15min":856.17700000000002},"last":2.681},{"interface":"front","average":{"1min":3.0579999999999998,"5min":12.347,"15min":6.0389999999999997},"min":{"1min":1.026,"5min":0.94899999999999995,"15min":0.94899999999999995},"max":{"1min":9.9260000000000002,"5min":503.36900000000003,"15min":503.36900000000003},"last":2.9239999999999999}]}]},{"osd":1,"up_from":15,"seq":64424509482,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2325524,"kb_used_data":2284812,"kb_used_omap":0,"kb_used_meta":40704,"kb_avail":92046316,"statfs":{"total":96636764160,"available":94255427584,"internally_reserved":0,"allocated":2339647488,"data_stored":2696888154,"data_compressed":37719632,"data_compressed_allocated":368033792,"data_compressed_original":736067584,"omap_allocated":0,"internal_metadata":41680896},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 28 10:29:53 2020","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.7350000000000001,"15min":1.8100000000000001},"min":{"1min":0.51300000000000001,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":2.7189999999999999,"5min":4.0819999999999999,"15min":4.0819999999999999},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.7430000000000001,"5min":2.2679999999999998,"15min":2.444},"min":{"1min":0.59399999999999997,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":2.9830000000000001,"5min":10.836,"15min":10.836},"last":2.9830000000000001}]},{"osd":2,"last update":"Mon Dec 28 10:29:53 2020","interfaces":[{"interface":"back","average":{"1min":1.756,"5min":12.887,"15min":5.359},"min":{"1min":0.66200000000000003,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":6.29,"5min":784.22199999999998,"15min":784.22199999999998},"last":1.538},{"interface":"front","average":{"1min":1.835,"5min":1.9590000000000001,"15min":1.9650000000000001},"min":{"1min":0.64600000000000002,"5min":0.625,"15min":0.625},"max":{"1min":3.819,"5min":12.396000000000001,"15min":12.396000000000001},"last":1.403}]},{"osd":3,"last update":"Mon Dec 28 10:29:53 2020","interfaces":[{"interface":"back","average":{"1min":61.206000000000003,"5min":40.906999999999996,"15min":45.015000000000001},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":507.279,"5min":753.41999999999996,"15min":753.41999999999996},"last":1.946},{"interface":"front","average":{"1min":41.82,"5min":10.255000000000001,"15min":5.0549999999999997},"min":{"1min":0.93200000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":507.13,"5min":507.13,"15min":507.13},"last":2.7759999999999998}]},{"osd":4,"last update":"Mon Dec 28 10:27:48 2020","interfaces":[{"interface":"back","average":{"1min":2.137,"5min":2.137,"15min":2.137},"min":{"1min":1.03,"5min":1.03,"15min":1.03},"max":{"1min":4.6180000000000003,"5min":4.6180000000000003,"15min":4.6180000000000003},"last":1.609},{"interface":"front","average":{"1min":2.3889999999999998,"5min":2.3889999999999998,"15min":2.3889999999999998},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":4.3860000000000001,"5min":4.3860000000000001,"15min":4.3860000000000001},"last":1.859}]},{"osd":5,"last update":"Mon Dec 28 10:29:53 2020","interfaces":[{"interface":"back","average":{"1min":3.3170000000000002,"5min":3.2160000000000002,"15min":3.2989999999999999},"min":{"1min":1.016,"5min":1.016,"15min":1.016},"max":{"1min":12.57,"5min":12.57,"15min":12.57},"last":1.665},{"interface":"front","average":{"1min":2.669,"5min":2.4340000000000002,"15min":2.3639999999999999},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":4.7140000000000004,"5min":11.34,"15min":11.34},"last":2.8460000000000001}]},{"osd":6,"last update":"Mon Dec 28 10:29:53 2020","interfaces":[{"interface":"back","average":{"1min":2.121,"5min":2.4420000000000002,"15min":2.5779999999999998},"min":{"1min":0.95699999999999996,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":4.0250000000000004,"5min":4.2519999999999998,"15min":4.2519999999999998},"last":2.028},{"interface":"front","average":{"1min":18.768000000000001,"5min":15.351000000000001,"15min":7.5529999999999999},"min":{"1min":1.131,"5min":1.0329999999999999,"15min":1.0329999999999999},"max":{"1min":344.22500000000002,"5min":757.19799999999998,"15min":757.19799999999998},"last":2.9119999999999999}]},{"osd":7,"last update":"Mon Dec 28 10:29:53 2020","interfaces":[{"interface":"back","average":{"1min":2.6160000000000001,"5min":3.0619999999999998,"15min":3.254},"min":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"max":{"1min":4.2939999999999996,"5min":8.8450000000000006,"15min":8.8450000000000006},"last":3.04},{"interface":"front","average":{"1min":2.5859999999999999,"5min":2.4990000000000001,"15min":2.46},"min":{"1min":1.2230000000000001,"5min":0.96599999999999997,"15min":0.96599999999999997},"max":{"1min":6.2569999999999997,"5min":11.372,"15min":11.372},"last":2.3860000000000001}]}]},{"osd":0,"up_from":15,"seq":64424509481,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4600336,"kb_used_data":4538568,"kb_used_omap":0,"kb_used_meta":61760,"kb_avail":89771504,"statfs":{"total":96636764160,"available":91926020096,"internally_reserved":0,"allocated":4647493632,"data_stored":5358985406,"data_compressed":75025437,"data_compressed_allocated":732143616,"data_compressed_original":1464287232,"omap_allocated":0,"internal_metadata":63242240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Dec 28 10:29:58 2020","interfaces":[{"interface":"back","average":{"1min":1.679,"5min":1.5649999999999999,"15min":1.548},"min":{"1min":0.65900000000000003,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":3.5800000000000001,"5min":3.9430000000000001,"15min":3.9430000000000001},"last":1.821},{"interface":"front","average":{"1min":1.9630000000000001,"5min":1.8089999999999999,"15min":1.653},"min":{"1min":0.88,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":3.3199999999999998,"5min":17.34,"15min":17.34},"last":1.8540000000000001}]},{"osd":2,"last update":"Mon Dec 28 10:29:58 2020","interfaces":[{"interface":"back","average":{"1min":46.095999999999997,"5min":28.902999999999999,"15min":31.105},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":527.29200000000003,"5min":527.29200000000003,"15min":527.29200000000003},"last":2.4449999999999998},{"interface":"front","average":{"1min":2.0059999999999998,"5min":1.831,"15min":1.7330000000000001},"min":{"1min":1.175,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":3.2250000000000001,"5min":8.7400000000000002,"15min":8.7400000000000002},"last":1.175}]},{"osd":3,"last update":"Mon Dec 28 10:29:58 2020","interfaces":[{"interface":"back","average":{"1min":51.545000000000002,"5min":11.798,"15min":5.1520000000000001},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":983.34400000000005,"5min":983.34400000000005,"15min":983.34400000000005},"last":2.8849999999999998},{"interface":"front","average":{"1min":12.106,"5min":4.4550000000000001,"15min":2.9079999999999999},"min":{"1min":1.103,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":187.756,"5min":187.756,"15min":187.756},"last":3.056}]},{"osd":4,"last update":"Mon Dec 28 10:27:52 2020","interfaces":[{"interface":"back","average":{"1min":1.974,"5min":1.974,"15min":1.974},"min":{"1min":1.095,"5min":1.095,"15min":1.095},"max":{"1min":2.8450000000000002,"5min":2.8450000000000002,"15min":2.8450000000000002},"last":7.1790000000000003},{"interface":"front","average":{"1min":2.3530000000000002,"5min":2.3530000000000002,"15min":2.3530000000000002},"min":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"max":{"1min":5.7830000000000004,"5min":5.7830000000000004,"15min":5.7830000000000004},"last":2.1579999999999999}]},{"osd":5,"last update":"Mon Dec 28 10:29:58 2020","interfaces":[{"interface":"back","average":{"1min":2.8330000000000002,"5min":5.1509999999999998,"15min":3.1349999999999998},"min":{"1min":1.385,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":5.9370000000000003,"5min":355.34500000000003,"15min":355.34500000000003},"last":3.2749999999999999},{"interface":"front","average":{"1min":2.7669999999999999,"5min":2.5899999999999999,"15min":2.3359999999999999},"min":{"1min":1.1080000000000001,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":9.8320000000000007,"5min":28.140000000000001,"15min":28.140000000000001},"last":1.6060000000000001}]},{"osd":6,"last update":"Mon Dec 28 10:29:58 2020","interfaces":[{"interface":"back","average":{"1min":2.4249999999999998,"5min":3.4740000000000002,"15min":2.8319999999999999},"min":{"1min":1.1319999999999999,"5min":0.88800000000000001,"15min":0.88800000000000001},"max":{"1min":3.9159999999999999,"5min":134.11600000000001,"15min":134.11600000000001},"last":2.419},{"interface":"front","average":{"1min":18.087,"5min":6.665,"15min":3.9359999999999999},"min":{"1min":1.73,"5min":0.95299999999999996,"15min":0.95299999999999996},"max":{"1min":225.87899999999999,"5min":225.87899999999999,"15min":225.87899999999999},"last":2.988}]},{"osd":7,"last update":"Mon Dec 28 10:29:58 2020","interfaces":[{"interface":"back","average":{"1min":2.754,"5min":2.472,"15min":2.4399999999999999},"min":{"1min":1.1259999999999999,"5min":1.008,"15min":1.008},"max":{"1min":4.5140000000000002,"5min":8.8529999999999998,"15min":8.8529999999999998},"last":2.6030000000000002},{"interface":"front","average":{"1min":2.3010000000000002,"5min":15.318,"15min":21.201000000000001},"min":{"1min":1.3,"5min":1.0840000000000001,"15min":1.0840000000000001},"max":{"1min":3.3919999999999999,"5min":371.392,"15min":371.392},"last":2.2269999999999999}]}]},{"osd":2,"up_from":15,"seq":64424509481,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1992600,"kb_used_data":1963984,"kb_used_omap":0,"kb_used_meta":28608,"kb_avail":92379240,"statfs":{"total":96636764160,"available":94596341760,"internally_reserved":0,"allocated":2011119616,"data_stored":2756233773,"data_compressed":76448272,"data_compressed_allocated":745623552,"data_compressed_original":1491247104,"omap_allocated":0,"internal_metadata":29294592},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 28 10:29:52 2020","interfaces":[{"interface":"back","average":{"1min":1.923,"5min":6.8739999999999997,"15min":8.9039999999999999},"min":{"1min":0.73799999999999999,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":3.6480000000000001,"5min":162.44499999999999,"15min":162.44499999999999},"last":1.4910000000000001},{"interface":"front","average":{"1min":2.1030000000000002,"5min":4.25,"15min":5.1200000000000001},"min":{"1min":0.78200000000000003,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":3.9620000000000002,"5min":76.317999999999998,"15min":76.317999999999998},"last":2.367}]},{"osd":1,"last update":"Mon Dec 28 10:29:52 2020","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":1.855,"15min":1.9610000000000001},"min":{"1min":0.65900000000000003,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":3.1739999999999999,"5min":5.6660000000000004,"15min":5.6660000000000004},"last":1.071},{"interface":"front","average":{"1min":2.0800000000000001,"5min":5.2389999999999999,"15min":3.1230000000000002},"min":{"1min":1.1519999999999999,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":4.3529999999999998,"5min":281.38299999999998,"15min":281.38299999999998},"last":1.5820000000000001}]},{"osd":3,"last update":"Mon Dec 28 10:29:55 2020","interfaces":[{"interface":"back","average":{"1min":1.9099999999999999,"5min":1.8149999999999999,"15min":1.7130000000000001},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":3.6579999999999999,"5min":4.6449999999999996,"15min":4.6449999999999996},"last":1.631},{"interface":"front","average":{"1min":1.806,"5min":2.0630000000000002,"15min":2.1629999999999998},"min":{"1min":0.85499999999999998,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":3.0960000000000001,"5min":5.7240000000000002,"15min":5.7240000000000002},"last":1.0009999999999999}]},{"osd":4,"last update":"Mon Dec 28 10:27:51 2020","interfaces":[{"interface":"back","average":{"1min":2.3250000000000002,"5min":2.3250000000000002,"15min":2.3250000000000002},"min":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"max":{"1min":5.6980000000000004,"5min":5.6980000000000004,"15min":5.6980000000000004},"last":2.3210000000000002},{"interface":"front","average":{"1min":2.3069999999999999,"5min":2.3069999999999999,"15min":2.3069999999999999},"min":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"max":{"1min":4.0730000000000004,"5min":4.0730000000000004,"15min":4.0730000000000004},"last":1.554}]},{"osd":5,"last update":"Mon Dec 28 10:29:52 2020","interfaces":[{"interface":"back","average":{"1min":23.582000000000001,"5min":6.5010000000000003,"15min":3.6800000000000002},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":365.40800000000002,"5min":365.40800000000002,"15min":365.40800000000002},"last":1.6819999999999999},{"interface":"front","average":{"1min":2.29,"5min":26.539999999999999,"15min":37.118000000000002},"min":{"1min":1.1910000000000001,"5min":1.028,"15min":1.028},"max":{"1min":4.8410000000000002,"5min":758.39099999999996,"15min":758.39099999999996},"last":1.7430000000000001}]},{"osd":6,"last update":"Mon Dec 28 10:29:52 2020","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.4359999999999999,"15min":2.5019999999999998},"min":{"1min":1.2929999999999999,"5min":0.95999999999999996,"15min":0.95999999999999996},"max":{"1min":4.0510000000000002,"5min":6.5700000000000003,"15min":6.5700000000000003},"last":2.2709999999999999},{"interface":"front","average":{"1min":2.4390000000000001,"5min":2.4239999999999999,"15min":2.4769999999999999},"min":{"1min":1.2450000000000001,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":4.359,"5min":6.4640000000000004,"15min":6.4640000000000004},"last":2.4089999999999998}]},{"osd":7,"last update":"Mon Dec 28 10:29:52 2020","interfaces":[{"interface":"back","average":{"1min":2.6859999999999999,"5min":2.7229999999999999,"15min":2.8559999999999999},"min":{"1min":1.637,"5min":1.002,"15min":1.002},"max":{"1min":4.4889999999999999,"5min":6.9870000000000001,"15min":6.9870000000000001},"last":2.448},{"interface":"front","average":{"1min":2.48,"5min":2.891,"15min":3.145},"min":{"1min":1.2869999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"max":{"1min":4.7290000000000001,"5min":6.8979999999999997,"15min":6.8979999999999997},"last":1.788}]}]},{"osd":3,"up_from":15,"seq":64424509481,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3321044,"kb_used_data":3279436,"kb_used_omap":0,"kb_used_meta":41600,"kb_avail":91050796,"statfs":{"total":96636764160,"available":93236015104,"internally_reserved":0,"allocated":3358142464,"data_stored":4099801574,"data_compressed":77073221,"data_compressed_allocated":752254976,"data_compressed_original":1504509952,"omap_allocated":0,"internal_metadata":42598400},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 28 10:29:57 2020","interfaces":[{"interface":"back","average":{"1min":2.0609999999999999,"5min":35.347000000000001,"15min":46.389000000000003},"min":{"1min":0.68200000000000005,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":4.0960000000000001,"5min":953.36000000000001,"15min":953.36000000000001},"last":4.0960000000000001},{"interface":"front","average":{"1min":1.665,"5min":1.8280000000000001,"15min":1.6819999999999999},"min":{"1min":0.67700000000000005,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":4.0780000000000003,"5min":12.409000000000001,"15min":12.409000000000001},"last":1.849}]},{"osd":1,"last update":"Mon Dec 28 10:29:57 2020","interfaces":[{"interface":"back","average":{"1min":1.7110000000000001,"5min":1.861,"15min":1.831},"min":{"1min":0.59299999999999997,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":2.7269999999999999,"5min":6.9219999999999997,"15min":6.9219999999999997},"last":2.5790000000000002},{"interface":"front","average":{"1min":2.2839999999999998,"5min":2.0710000000000002,"15min":2.085},"min":{"1min":1.0029999999999999,"5min":0.629,"15min":0.629},"max":{"1min":4.1630000000000003,"5min":6.3739999999999997,"15min":6.3739999999999997},"last":3.1419999999999999}]},{"osd":2,"last update":"Mon Dec 28 10:29:57 2020","interfaces":[{"interface":"back","average":{"1min":2.0800000000000001,"5min":2.0899999999999999,"15min":2.1000000000000001},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":4.0220000000000002,"5min":5.7800000000000002,"15min":5.7800000000000002},"last":3.8210000000000002},{"interface":"front","average":{"1min":2.7040000000000002,"5min":2.2440000000000002,"15min":2.27},"min":{"1min":1.0429999999999999,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":4.117,"5min":4.8890000000000002,"15min":4.8890000000000002},"last":3.2040000000000002}]},{"osd":4,"last update":"Mon Dec 28 10:27:50 2020","interfaces":[{"interface":"back","average":{"1min":2.6549999999999998,"5min":2.6549999999999998,"15min":2.6549999999999998},"min":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"max":{"1min":5.1959999999999997,"5min":5.1959999999999997,"15min":5.1959999999999997},"last":2.8359999999999999},{"interface":"front","average":{"1min":2.7509999999999999,"5min":2.7509999999999999,"15min":2.7509999999999999},"min":{"1min":1.262,"5min":1.262,"15min":1.262},"max":{"1min":4.7080000000000002,"5min":4.7080000000000002,"15min":4.7080000000000002},"last":2.9950000000000001}]},{"osd":5,"last update":"Mon Dec 28 10:29:57 2020","interfaces":[{"interface":"back","average":{"1min":2.3879999999999999,"5min":2.806,"15min":2.883},"min":{"1min":0.93799999999999994,"5min":0.874,"15min":0.874},"max":{"1min":4.1550000000000002,"5min":10.362,"15min":10.362},"last":4.1550000000000002},{"interface":"front","average":{"1min":2.9710000000000001,"5min":2.887,"15min":2.8620000000000001},"min":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"max":{"1min":4.9580000000000002,"5min":11.971,"15min":11.971},"last":3.0790000000000002}]},{"osd":6,"last update":"Mon Dec 28 10:29:57 2020","interfaces":[{"interface":"back","average":{"1min":13.807,"5min":13.57,"15min":6.4790000000000001},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":180.655,"5min":813.024,"15min":813.024},"last":3.9910000000000001},{"interface":"front","average":{"1min":2.5070000000000001,"5min":2.6349999999999998,"15min":2.7290000000000001},"min":{"1min":1.044,"5min":1,"15min":1},"max":{"1min":3.9220000000000002,"5min":6.2750000000000004,"15min":6.2750000000000004},"last":3.9220000000000002}]},{"osd":7,"last update":"Mon Dec 28 10:29:57 2020","interfaces":[{"interface":"back","average":{"1min":3.052,"5min":2.7029999999999998,"15min":2.7749999999999999},"min":{"1min":1.3149999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"max":{"1min":5.0229999999999997,"5min":5.984,"15min":5.984},"last":3.3119999999999998},{"interface":"front","average":{"1min":2.778,"5min":2.8980000000000001,"15min":3.056},"min":{"1min":1.075,"5min":1.075,"15min":1.075},"max":{"1min":4.6769999999999996,"5min":6.2729999999999997,"15min":6.2729999999999997},"last":4.6769999999999996}]}]},{"osd":4,"up_from":94,"seq":403726925835,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3974984,"kb_used_data":3926440,"kb_used_omap":0,"kb_used_meta":37631,"kb_avail":90396856,"statfs":{"total":96636764160,"available":92566380544,"internally_reserved":0,"allocated":4020674560,"data_stored":4730188751,"data_compressed":74266497,"data_compressed_allocated":725573632,"data_compressed_original":1451147264,"omap_allocated":964,"internal_metadata":38534204},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.204}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.484}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.53}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6020000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0819999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.6219999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0470000000000002}]}]},{"osd":5,"up_from":15,"seq":64424509482,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7909896,"kb_used_data":7815680,"kb_used_omap":0,"kb_used_meta":94208,"kb_avail":86461944,"statfs":{"total":96636764160,"available":88537030656,"internally_reserved":0,"allocated":8003256320,"data_stored":9444818699,"data_compressed":150749824,"data_compressed_allocated":1472462848,"data_compressed_original":2944925696,"omap_allocated":0,"internal_metadata":96468992},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Dec 28 10:29:53 2020","interfaces":[{"interface":"back","average":{"1min":2.0630000000000002,"5min":4.2270000000000003,"15min":2.8220000000000001},"min":{"1min":0.72299999999999998,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":4.4500000000000002,"5min":168.51599999999999,"15min":168.51599999999999},"last":3.1419999999999999},{"interface":"front","average":{"1min":2.3260000000000001,"5min":4.2729999999999997,"15min":2.8039999999999998},"min":{"1min":0.76700000000000002,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":4.4880000000000004,"5min":168.56200000000001,"15min":168.56200000000001},"last":3.2629999999999999}]},{"osd":1,"last update":"Mon Dec 28 10:29:53 2020","interfaces":[{"interface":"back","average":{"1min":31.669,"5min":7.9210000000000003,"15min":3.774},"min":{"1min":1.0780000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":504.11000000000001,"5min":504.11000000000001,"15min":504.11000000000001},"last":3.0680000000000001},{"interface":"front","average":{"1min":31.68,"5min":18.266999999999999,"15min":7.3570000000000002},"min":{"1min":0.79900000000000004,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":504.06900000000002,"5min":870.06200000000001,"15min":870.06200000000001},"last":1.6819999999999999}]},{"osd":2,"last update":"Mon Dec 28 10:29:53 2020","interfaces":[{"interface":"back","average":{"1min":87.649000000000001,"5min":19.332000000000001,"15min":7.8899999999999997},"min":{"1min":0.87,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":985.255,"5min":985.255,"15min":985.255},"last":1.4990000000000001},{"interface":"front","average":{"1min":3.02,"5min":2.4769999999999999,"15min":2.2909999999999999},"min":{"1min":1.2210000000000001,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":13.564,"5min":13.564,"15min":13.564},"last":1.6240000000000001}]},{"osd":3,"last update":"Mon Dec 28 10:29:53 2020","interfaces":[{"interface":"back","average":{"1min":2.3679999999999999,"5min":2.3719999999999999,"15min":2.254},"min":{"1min":1.014,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":3.7770000000000001,"5min":11.653,"15min":11.653},"last":3.3239999999999998},{"interface":"front","average":{"1min":2.6800000000000002,"5min":2.665,"15min":2.5609999999999999},"min":{"1min":1.089,"5min":0.90000000000000002,"15min":0.90000000000000002},"max":{"1min":8.8889999999999993,"5min":10.91,"15min":10.91},"last":2.0499999999999998}]},{"osd":4,"last update":"Mon Dec 28 10:27:50 2020","interfaces":[{"interface":"back","average":{"1min":2.726,"5min":2.726,"15min":2.726},"min":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"max":{"1min":5.601,"5min":5.601,"15min":5.601},"last":3.2000000000000002},{"interface":"front","average":{"1min":2.5390000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"min":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"max":{"1min":5.8019999999999996,"5min":5.8019999999999996,"15min":5.8019999999999996},"last":2.0049999999999999}]},{"osd":6,"last update":"Mon Dec 28 10:29:53 2020","interfaces":[{"interface":"back","average":{"1min":2.5449999999999999,"5min":2.7839999999999998,"15min":2.7709999999999999},"min":{"1min":1.0449999999999999,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":10.602,"5min":13.284000000000001,"15min":13.284000000000001},"last":1.5720000000000001},{"interface":"front","average":{"1min":3.2909999999999999,"5min":6.8470000000000004,"15min":4.032},"min":{"1min":1.1339999999999999,"5min":1.044,"15min":1.044},"max":{"1min":13.528,"5min":335.69,"15min":335.69},"last":1.9670000000000001}]},{"osd":7,"last update":"Mon Dec 28 10:28:58 2020","interfaces":[{"interface":"back","average":{"1min":3.1899999999999999,"5min":8.5540000000000003,"15min":9.4480000000000004},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":11.612,"5min":163.16399999999999,"15min":163.16399999999999},"last":3.3799999999999999},{"interface":"front","average":{"1min":42.051000000000002,"5min":10.766,"15min":5.5519999999999996},"min":{"1min":1.2370000000000001,"5min":1.069,"15min":1.069},"max":{"1min":661.77800000000002,"5min":661.77800000000002,"15min":661.77800000000002},"last":2.1019999999999999}]}]}]}} , 2020-12-28T10:30:05.024+0000 7f108c9d0700 -1 osd.3 pg_epoch: 120 pg[3.2s0( v 91'925 (0'0,91'925] local-lis/les=118/119 n=923 ec=23/23 lis/c=118/118 les/c/f=119/119/0 sis=118) [3,5,6,4]p3(0) r=0 lpr=118 crt=91'925 lcod 91'924 mlcod 0'0 active+clean] scrubber pg(3.2s0) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6(2) (0x560d16fbf5a0) 2020-12-29T00:05:30.347 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-28T10:30:35.025 INFO:tasks.ceph.ceph_manager.ceph:recovered! , 2020-12-28T10:30:35.026 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes , 2020-12-28T10:30:35.356 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2020-12-29T00:05:30.348 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-28T10:30:41.751 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2020-12-28T10:28:14.506230+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=28, tm_hour=10, tm_min=28, tm_sec=14, tm_wday=0, tm_yday=363, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=28, tm_hour=10, tm_min=30, tm_sec=36, tm_wday=0, tm_yday=363, tm_isdst=0) , 2020-12-28T10:30:41.752 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2020-12-28T10:28:38.540092+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=28, tm_hour=10, tm_min=28, tm_sec=38, tm_wday=0, tm_yday=363, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=28, tm_hour=10, tm_min=30, tm_sec=36, tm_wday=0, tm_yday=363, tm_isdst=0) , 2020-12-28T10:30:41.752 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2020-12-28T10:30:24.640833+0000 time.struct_time(tm_year=2020, tm_mon=12, tm_mday=28, tm_hour=10, tm_min=30, tm_sec=24, tm_wday=0, tm_yday=363, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=12, tm_mday=28, tm_hour=10, tm_min=30, tm_sec=36, tm_wday=0, tm_yday=363, tm_isdst=0) 2020-12-29T00:05:30.350 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-28T10:32:01.076+0000 7fbf94d65700 -1 osd.0 pg_epoch: 134 pg[4.3( v 130'8038 (78'5037,130'8038] local-lis/les=26/27 n=986 ec=26/26 lis/c=26/26 les/c/f=27/27/0 sis=26) [0,5,7] r=0 lpr=26 crt=130'8038 lcod 130'8037 mlcod 130'8037 active+clean] scrubber pg(4.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55b922c946e0) , 2020-12-28T10:32:02.048+0000 7fbf90d5d700 -1 osd.0 pg_epoch: 134 pg[4.3( v 130'8038 (78'5037,130'8038] local-lis/les=26/27 n=986 ec=26/26 lis/c=26/26 les/c/f=27/27/0 sis=26) [0,5,7] r=0 lpr=26 crt=130'8038 lcod 130'8037 mlcod 130'8037 active+clean] scrubber pg(4.3) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x55b92bd01ce0) , 2020-12-28T10:32:02.120+0000 7f44eeebf700 -1 osd.2 pg_epoch: 134 pg[2.10( empty local-lis/les=102/103 n=0 ec=63/18 lis/c=102/102 les/c/f=103/103/0 sis=102) [2,1,0] r=0 lpr=102 crt=0'0 mlcod 0'0 active+clean] scrubber pg(2.10) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 0 (0x556272404c60) 2020-12-29T00:05:30.350 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-28T10:32:52.019 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi092.front.sepia.ceph.com , 2020-12-28T10:32:52.019 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 , 2020-12-28T10:32:52.218 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2020-12-29T00:05:30.351 WARNING:scrape:Ignoring malparsed backtrace: 2020-12-28T10:37:41.506 INFO:teuthology.orchestra.run.smithi016.stdout:============================================================================== , 2020-12-28T10:37:41.506 INFO:teuthology.orchestra.run.smithi016.stdout:-hv01.front.sepi 216.240.36.24 3 u 19 64 377 0.095 0.915 1.676 , 2020-12-28T10:37:41.506 INFO:teuthology.orchestra.run.smithi016.stdout:*hv02.front.sepi 163.237.218.18 2 u 21 64 377 0.095 0.059 1.985 2020-12-29T00:05:42.483 INFO:scrape:Found 11 distinct failure reasons 2020-12-29T00:05:42.484 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2020-12-29T00:05:42.485 INFO:scrape:2 jobs: ['5742346', '5742190'] 2020-12-29T00:05:42.485 INFO:scrape:suites intersection: ['msgr/async', 'rados'] 2020-12-29T00:05:42.486 INFO:scrape:suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/cache-agent-big}'] 2020-12-29T00:05:42.486 INFO:scrape: 2020-12-29T00:05:42.487 INFO:scrape:Dead: b'2020-12-28T21:55:52.665 DEBUG:teuthology.exit:Finished running handlers' 2020-12-29T00:05:42.487 INFO:scrape:6 jobs: ['5742162', '5742120', '5742156', '5742293', '5742212', '5742014'] 2020-12-29T00:05:42.487 INFO:scrape:suites intersection: ['thrashosds-health'] 2020-12-29T00:05:42.488 INFO:scrape:suites union: ['1-install/nautilus', '1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-active-recovery}', '2-recovery-overrides/{more-partial-recovery}', '3-thrash/default', '4-workload/{rbd-cls', '5-workload/{radosbench', '6-finish-upgrade', '7-octopus', '8-workload/{rbd-python', 'backoff/peering_and_degraded', 'bluestore-bitmap', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-hybrid', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack', 'rbd-import-export', 'rbd_api}', 'readwrite', 'recovery-overrides/{default}', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashers/pggrow', 'thrashosds-health', 'ubuntu_latest}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/rados_api_tests}', 'workloads/radosbench-high-concurrency}', 'workloads/small-objects}', 'workloads/snaps-few-objects}'] 2020-12-29T00:05:42.488 INFO:scrape: 2020-12-29T00:05:42.489 INFO:scrape:Failure: Command failed (workunit test cephadm/test_adoption.sh) on smithi071 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ce2ba5cb8764cac5bd77397fd0c120a799451956 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' 2020-12-29T00:05:42.489 INFO:scrape:1 jobs: ['5742080'] 2020-12-29T00:05:42.490 INFO:scrape:suites: ['mon_election/connectivity', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_adoption}'] 2020-12-29T00:05:42.490 INFO:scrape: 2020-12-29T00:05:42.490 INFO:scrape:Failure: unexpected end of data 2020-12-29T00:05:42.491 INFO:scrape:1 jobs: ['5742043'] 2020-12-29T00:05:42.491 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/none', 'thrashosds-health', 'workloads/radosbench}'] 2020-12-29T00:05:42.492 INFO:scrape: 2020-12-29T00:05:42.492 INFO:scrape:Failure: reached maximum tries (180) after waiting for 180 seconds 2020-12-29T00:05:42.492 INFO:scrape:5 jobs: ['5742356', '5742172', '5742261', '5742084', '5742228'] 2020-12-29T00:05:42.493 INFO:scrape:suites intersection: ['3-start-upgrade', '4-wait', 'fixed-2', 'rados/cephadm/upgrade/{1-start'] 2020-12-29T00:05:42.493 INFO:scrape:suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'distro$/{rhel_8.0}', 'distro$/{rhel_latest}', 'fixed-2', 'mon_election/classic}', 'mon_election/connectivity}', 'rados/cephadm/upgrade/{1-start'] 2020-12-29T00:05:42.494 INFO:scrape: 2020-12-29T00:05:42.494 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/rados/test.sh 2020-12-29T00:05:42.495 INFO:scrape:2 jobs: ['5742053', '5742272'] 2020-12-29T00:05:42.495 INFO:scrape:suites intersection: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}'] 2020-12-29T00:05:42.495 INFO:scrape:suites union: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}'] 2020-12-29T00:05:42.496 INFO:scrape: 2020-12-29T00:05:42.496 INFO:scrape:Failure: "2020-12-28T10:37:58.458616+0000 mon.a (mon.0) 21 : cluster [WRN] Health check failed: 1/9 mons down, quorum a,b,c,d,e,f,g,h (MON_DOWN)" in cluster log 2020-12-29T00:05:42.497 INFO:scrape:1 jobs: ['5742201'] 2020-12-29T00:05:42.497 INFO:scrape:suites: ['mon_election/connectivity', 'msgr-failures/many', 'msgr/async-v1only', 'no_pools', 'objectstore/bluestore-low-osd-mem-target', 'rados', 'rados/multimon/{clusters/9', 'supported-random-distro$/{rhel_8}', 'tasks/mon_clock_no_skews}'] 2020-12-29T00:05:42.497 INFO:scrape: 2020-12-29T00:05:42.498 INFO:scrape:Failure: psutil.NoSuchProcess process no longer exists (pid=5756) 2020-12-29T00:05:42.498 INFO:scrape:1 jobs: ['5742361'] 2020-12-29T00:05:42.499 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/small-objects-balanced}'] 2020-12-29T00:05:42.499 INFO:scrape: 2020-12-29T00:05:42.499 INFO:scrape:Failure: Exiting scrub checking -- not all pgs scrubbed. 2020-12-29T00:05:42.500 INFO:scrape:1 jobs: ['5742222'] 2020-12-29T00:05:42.500 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-partial-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/cache-pool-snaps}'] 2020-12-29T00:05:42.501 INFO:scrape: 2020-12-29T00:05:42.501 INFO:scrape:Failure: Command failed (workunit test rados/test.sh) on smithi099 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ce2ba5cb8764cac5bd77397fd0c120a799451956 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 2020-12-29T00:05:42.501 INFO:scrape:1 jobs: ['5742284'] 2020-12-29T00:05:42.502 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/default', 'thrashosds-health', 'workloads/rados_api_tests}'] 2020-12-29T00:05:42.502 INFO:scrape: 2020-12-29T00:05:42.503 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi080 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ce2ba5cb8764cac5bd77397fd0c120a799451956 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2020-12-29T00:05:42.503 INFO:scrape:1 jobs: ['5742254'] 2020-12-29T00:05:42.504 INFO:scrape:suites: ['mon_election/connectivity', 'rados/cephadm/workunits/{distro/ubuntu_18.04_podman', 'task/test_cephadm}'] 2020-12-29T00:05:42.504 INFO:scrape: 2020-12-29T00:05:42.516 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/ failed: 15 dead: 7 running: 0 waiting: 0 queued: 0 passed: 345 Fail ================================================================= [5742043] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 01:55:31 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742043/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742043/ unexpected end of data [5742053] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 06:52:54 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742053/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742053/ sentry: https://sentry.ceph.com/organizations/ceph/?query=051e6090993444d7b5bd2c80923cbea9 Command failed (workunit test rados/test.sh) on smithi169 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ce2ba5cb8764cac5bd77397fd0c120a799451956 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 ALLOW_TIMEOUTS=1 adjust- ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5742080] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_adoption} ----------------------------------------------------------------- time: 00:14:23 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742080/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742080/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7b276db720234e1b8e892c1a77a117c8 Command failed (workunit test cephadm/test_adoption.sh) on smithi071 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ce2ba5cb8764cac5bd77397fd0c120a799451956 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_adoption.sh' [5742084] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{rhel_latest} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:16:28 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742084/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742084/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8f6147eb06024c26a67005b38aa090f6 reached maximum tries (180) after waiting for 180 seconds [5742172] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{rhel_8.0} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:19:22 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742172/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742172/ sentry: https://sentry.ceph.com/organizations/ceph/?query=03c7d06366df4fba83e901777cae50de reached maximum tries (180) after waiting for 180 seconds [5742190] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache-agent-big} ----------------------------------------------------------------- time: 00:15:37 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742190/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742190/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 10000 --objects 6600 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 1200 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' [5742201] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} tasks/mon_clock_no_skews} ----------------------------------------------------------------- time: 00:08:51 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742201/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742201/ "2020-12-28T10:37:58.458616+0000 mon.a (mon.0) 21 : cluster [WRN] Health check failed: 1/9 mons down, quorum a,b,c,d,e,f,g,h (MON_DOWN)" in cluster log [5742222] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps} ----------------------------------------------------------------- time: 00:36:50 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742222/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742222/ Exiting scrub checking -- not all pgs scrubbed. [5742228] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{rhel_latest} fixed-2 mon_election/classic} ----------------------------------------------------------------- time: 00:16:36 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742228/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742228/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d466bb7f0dac4789862849f4f1dc88aa reached maximum tries (180) after waiting for 180 seconds [5742254] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:14:04 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742254/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742254/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1905763e5baf46bda1d1506993a1f16b Command failed (workunit test cephadm/test_cephadm.sh) on smithi080 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ce2ba5cb8764cac5bd77397fd0c120a799451956 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [5742261] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{rhel_latest} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:16:51 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742261/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742261/ sentry: https://sentry.ceph.com/organizations/ceph/?query=df976eb312a340e4a4d42496c7c982e2 reached maximum tries (180) after waiting for 180 seconds [5742272] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async objectstore/filestore-xfs rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 06:46:43 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742272/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742272/ sentry: https://sentry.ceph.com/organizations/ceph/?query=510ffb2b757f40d39dfe44416b8cd344 Command failed (workunit test rados/test.sh) on smithi183 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ce2ba5cb8764cac5bd77397fd0c120a799451956 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 ALLOW_TIMEOUTS=1 adjust- ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5742284] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:32:50 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742284/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742284/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7e6b4154eb074397b64cda56cb3ba132 Command failed (workunit test rados/test.sh) on smithi099 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ce2ba5cb8764cac5bd77397fd0c120a799451956 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [5742346] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:10:48 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742346/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742346/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' [5742356] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{rhel_latest} fixed-2 mon_election/connectivity} ----------------------------------------------------------------- time: 00:16:59 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742356/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742356/ sentry: https://sentry.ceph.com/organizations/ceph/?query=21e518a9e4054210a71b529249432c5c reached maximum tries (180) after waiting for 180 seconds Dead ================================================================= [5742014] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742014/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742014/ [5742120] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742120/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742120/ [5742156] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742156/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742156/ [5742162] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742162/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742162/ [5742212] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap mon_election/connectivity thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742212/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742212/ [5742293] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/radosbench-high-concurrency} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742293/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742293/ [5742361] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/small-objects-balanced} ----------------------------------------------------------------- time: 11:41:03 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742361/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742361/ psutil.NoSuchProcess process no longer exists (pid=5756) Pass ================================================================= [5742011] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:25:21 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742011/ [5742012] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados tasks/mon_recovery validater/lockdep} time: 00:12:04 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742012/ [5742013] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:24:23 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742013/ [5742015] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} tasks/mon_clock_with_skews} time: 00:08:28 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742015/ [5742016] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:28:48 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742016/ [5742017] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/cosbench_64K_write} time: 00:25:53 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742017/ [5742018] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/force-sync-many workloads/rados_mon_workunits} time: 00:33:46 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742018/ [5742019] rados/cephadm/with-work/{distro/rhel_8.0 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} time: 00:42:27 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742019/ [5742020] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:26:44 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742020/ [5742021] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/rados_cls_all} time: 00:24:10 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742021/ [5742022] rados/singleton/{all/deduptool mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:12:31 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742022/ [5742023] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:42:16 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742023/ [5742024] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:08:02 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742024/ [5742025] rados/objectstore/{backends/alloc-hint supported-random-distro$/{centos_8}} time: 00:15:48 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742025/ [5742026] rados/rest/{mgr-restful supported-random-distro$/{rhel_8}} time: 00:17:32 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742026/ [5742027] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:14:54 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742027/ [5742028] rados/standalone/{mon_election/classic supported-random-distro$/{rhel_8} workloads/crush} time: 00:19:22 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742028/ [5742029] rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap mon_election/classic thrashosds-health ubuntu_latest} time: 05:41:40 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742029/ [5742030] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} time: 00:19:13 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742030/ [5742031] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/careful thrashosds-health workloads/snaps-few-objects} time: 00:33:53 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742031/ [5742032] rados/cephadm/dashboard/{distro/centos_latest task/test_e2e} time: 00:32:06 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742032/ [5742033] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:08 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742033/ [5742034] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/radosbench-high-concurrency} time: 01:05:27 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742034/ [5742035] rados/cephadm/orchestrator_cli/{2-node-mgr orchestrator_cli supported-random-distro$/{ubuntu_latest}} time: 00:09:36 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742035/ [5742036] rados/singleton/{all/divergent_priors mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:11:04 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742036/ [5742037] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:13:31 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742037/ [5742038] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/classic start} time: 00:13:43 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742038/ [5742039] rados/cephadm/smoke-roleless/{distro/centos_8.0 start} time: 00:13:02 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742039/ [5742040] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-snappy supported-random-distro$/{rhel_8} tasks/progress} time: 00:24:11 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742040/ [5742041] rados/singleton/{all/divergent_priors2 mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:10:05 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742041/ [5742042] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:07:41 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742042/ [5742044] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:10:05 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742044/ [5742045] rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{ubuntu_18.04_podman} fixed-2 mon_election/classic} time: 00:18:32 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742045/ [5742046] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:10:27 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742046/ [5742047] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_orch_cli} time: 00:18:57 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742047/ [5742048] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/erasure-code} time: 00:45:20 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742048/ [5742049] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/connectivity start} time: 00:12:52 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742049/ [5742050] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/rados_python} time: 00:19:13 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742050/ [5742051] rados/singleton/{all/dump-stuck mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} time: 00:10:26 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742051/ [5742052] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:12:04 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742052/ [5742054] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:28:26 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742054/ [5742055] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/redirect} time: 00:15:14 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742055/ [5742056] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:12:45 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742056/ [5742057] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/default thrashosds-health workloads/test_rbd_api} time: 00:17:34 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742057/ [5742058] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} tasks/mon_recovery} time: 00:16:08 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742058/ [5742059] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-radosbench} time: 02:30:46 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742059/ [5742060] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/snaps-few-objects} time: 00:32:51 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742060/ [5742061] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} time: 00:29:05 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742061/ [5742062] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/classic start} time: 00:14:22 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742062/ [5742063] rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:41:34 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742063/ [5742064] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8} tasks/dashboard} time: 01:08:48 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742064/ [5742065] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{ubuntu_latest}} time: 00:09:03 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742065/ [5742066] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:42 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742066/ [5742067] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:15:18 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742067/ [5742068] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/redirect_promote_tests} time: 00:18:49 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742068/ [5742069] rados/cephadm/smoke-roleless/{distro/centos_latest start} time: 00:12:23 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742069/ [5742070] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:12:35 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742070/ [5742071] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/mapgap thrashosds-health workloads/cache-snaps} time: 00:29:45 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742071/ [5742072] rados/singleton/{all/erasure-code-nonregression mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}} time: 00:14:56 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742072/ [5742073] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:27:55 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742073/ [5742074] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/connectivity start} time: 00:12:08 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742074/ [5742075] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} tasks/prometheus} time: 00:08:29 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742075/ [5742076] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/mgr} time: 00:18:37 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742076/ [5742077] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:09:12 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742077/ [5742078] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/redirect_set_object} time: 00:13:22 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742078/ [5742079] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} time: 00:38:12 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742079/ [5742081] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} tasks/rados_stress_watch} time: 00:24:06 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742081/ [5742082] rados/singleton/{all/lost-unfound-delete mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:52:04 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742082/ [5742083] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:05:59 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742083/ [5742085] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:08:29 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742085/ [5742086] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/set-chunks-read} time: 00:13:04 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742086/ [5742087] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/classic start} time: 00:11:27 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742087/ [5742088] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:07:58 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742088/ [5742089] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:28:05 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742089/ [5742090] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados tasks/rados_cls_all validater/lockdep} time: 00:28:25 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742090/ [5742091] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:25:48 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742091/ [5742092] rados/cephadm/smoke-roleless/{distro/rhel_8.0 start} time: 00:14:07 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742092/ [5742093] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews} time: 00:08:26 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742093/ [5742094] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-small-objects-balanced} time: 00:27:17 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742094/ [5742095] rados/singleton/{all/lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:37:11 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742095/ [5742096] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/one workloads/pool-create-delete} time: 00:17:28 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742096/ [5742097] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/morepggrow thrashosds-health workloads/radosbench} time: 01:28:23 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742097/ [5742098] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:32 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742098/ [5742099] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:17:38 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742099/ [5742100] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/small-objects-balanced} time: 00:21:16 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742100/ [5742101] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/connectivity start} time: 00:12:10 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742101/ [5742102] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/misc} time: 00:32:11 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742102/ [5742103] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} time: 00:04:51 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742103/ [5742104] rados/cephadm/with-work/{distro/ubuntu_18.04_podman fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} time: 00:29:01 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742104/ [5742105] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:06:41 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742105/ [5742106] rados/objectstore/{backends/filejournal supported-random-distro$/{ubuntu_latest}} time: 00:05:24 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742106/ [5742107] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/none thrashosds-health workloads/rbd_cls} time: 00:15:08 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742107/ [5742108] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:13:01 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742108/ [5742109] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} tasks/workunits} time: 00:16:18 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742109/ [5742110] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/classic start} time: 00:11:38 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742110/ [5742111] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/small-objects-localized} time: 00:20:29 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742111/ [5742112] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:08:15 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742112/ [5742113] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} tasks/rados_striper} time: 00:10:17 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742113/ [5742114] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} time: 00:14:17 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742114/ [5742115] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:42:47 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742115/ [5742116] rados/cephadm/smoke-roleless/{distro/rhel_latest start} time: 00:12:10 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742116/ [5742117] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_cephadm} time: 00:21:48 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742117/ [5742118] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:06:29 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742118/ [5742119] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/connectivity start} time: 00:13:29 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742119/ [5742121] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:13:40 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742121/ [5742122] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:09:46 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742122/ [5742123] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:22:05 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742123/ [5742124] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:11:33 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742124/ [5742125] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados tasks/mon_recovery validater/valgrind} time: 00:28:57 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742125/ [5742126] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{ubuntu_18.04} fixed-2 mon_election/classic} time: 00:16:22 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742126/ [5742127] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:56 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742127/ [5742128] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/mon} time: 00:44:26 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742128/ [5742129] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:04:45 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742129/ [5742130] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read} time: 00:24:30 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742130/ [5742131] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:28:44 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742131/ [5742132] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:17:32 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742132/ [5742133] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/sync-many workloads/rados_5925} time: 00:11:25 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742133/ [5742134] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:35 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742134/ [5742135] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/pggrow thrashosds-health workloads/snaps-few-objects} time: 00:34:51 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742135/ [5742136] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/snaps-few-objects-balanced} time: 00:26:42 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742136/ [5742137] rados/singleton/{all/mon-auth-caps mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 00:17:13 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742137/ [5742138] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:08:34 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742138/ [5742139] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/classic start} time: 00:12:50 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742139/ [5742140] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:14:42 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742140/ [5742141] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04 start} time: 00:11:11 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742141/ [5742142] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} tasks/rados_workunit_loadgen_big} time: 00:39:08 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742142/ [5742143] rados/singleton/{all/mon-config-key-caps mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} time: 00:18:23 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742143/ [5742144] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-hybrid supported-random-distro$/{centos_8} tasks/crash} time: 00:11:18 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742144/ [5742145] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} tasks/dashboard} time: 01:09:14 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742145/ [5742146] rados/objectstore/{backends/filestore-idempotent-aio-journal supported-random-distro$/{rhel_8}} time: 02:20:22 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742146/ [5742147] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} time: 00:20:06 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742147/ [5742148] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} time: 00:18:12 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742148/ [5742149] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/snaps-few-objects-localized} time: 00:23:53 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742149/ [5742150] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_cephadm_repos} time: 00:10:59 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742150/ [5742151] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:21:29 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742151/ [5742152] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} time: 00:38:51 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742152/ [5742153] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/osd} time: 03:12:20 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742153/ [5742154] rados/singleton/{all/mon-config-keys mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:17:56 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742154/ [5742155] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/connectivity start} time: 00:14:58 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742155/ [5742157] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:10:38 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742157/ [5742158] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/classic start} time: 00:14:25 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742158/ [5742159] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:28:12 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742159/ [5742160] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados tasks/rados_api_tests validater/lockdep} time: 00:31:34 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742160/ [5742161] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:18:52 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742161/ [5742163] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04_podman start} time: 00:18:34 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742163/ [5742164] rados/singleton/{all/mon-config mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} time: 00:17:02 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742164/ [5742165] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-stupid supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:08:14 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742165/ [5742166] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:31 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742166/ [5742167] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:09:23 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742167/ [5742168] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:21:53 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742168/ [5742169] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_api_tests} time: 00:29:52 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742169/ [5742170] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/careful thrashosds-health workloads/cache-snaps} time: 00:29:04 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742170/ [5742171] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/write_fadvise_dontneed} time: 00:16:23 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742171/ [5742173] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:16:31 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742173/ [5742174] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} tasks/rados_workunit_loadgen_mix} time: 00:27:28 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742174/ [5742175] rados/singleton/{all/osd-backfill mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:18:49 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742175/ [5742176] rados/cephadm/with-work/{distro/rhel_8.0 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_python} time: 00:31:43 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742176/ [5742177] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-overwrites} time: 00:21:43 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742177/ [5742178] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/connectivity start} time: 00:11:38 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742178/ [5742179] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} tasks/failover} time: 00:10:05 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742179/ [5742180] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/scrub} time: 01:45:28 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742180/ [5742181] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/admin_socket_objecter_requests} time: 00:25:51 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742181/ [5742182] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:08:47 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742182/ [5742183] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_orch_cli} time: 00:19:10 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742183/ [5742184] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:29:24 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742184/ [5742185] rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:20:04 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742185/ [5742186] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/filestore-xfs supported-random-distro$/{centos_8} tasks/dashboard} time: 01:08:14 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742186/ [5742187] rados/objectstore/{backends/filestore-idempotent supported-random-distro$/{ubuntu_latest}} time: 02:26:11 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742187/ [5742188] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/radosbench} time: 01:32:48 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742188/ [5742189] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/classic start} time: 00:12:12 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742189/ [5742191] rados/cephadm/smoke-roleless/{distro/ubuntu_latest start} time: 00:11:02 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742191/ [5742192] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:13:16 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742192/ [5742193] rados/singleton/{all/osd-recovery mon_election/connectivity msgr-failures/many msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8}} time: 00:13:25 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742193/ [5742194] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} time: 00:40:04 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742194/ [5742195] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:13:26 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742195/ [5742196] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados tasks/rados_cls_all validater/valgrind} time: 02:15:27 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742196/ [5742197] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:30:12 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742197/ [5742198] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:32 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742198/ [5742199] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/connectivity start} time: 00:11:35 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742199/ [5742200] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/cache-agent-small} time: 00:15:40 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742200/ [5742202] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects} time: 00:17:37 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742202/ [5742203] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:09:07 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742203/ [5742204] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/force-sync-many workloads/rados_mon_osdmap_prune} time: 00:55:19 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742204/ [5742205] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} tasks/rados_workunit_loadgen_mostlyread} time: 00:39:20 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742205/ [5742206] rados/singleton/{all/peer mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} time: 00:09:39 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742206/ [5742207] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/rbd_cls} time: 00:18:52 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742207/ [5742208] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 00:46:30 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742208/ [5742209] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:06:36 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742209/ [5742210] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:14:44 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742210/ [5742211] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/crush} time: 00:15:10 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742211/ [5742213] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_adoption} time: 00:16:36 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742213/ [5742214] rados/cephadm/dashboard/{distro/centos_latest task/test_e2e} time: 00:32:11 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742214/ [5742215] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/cache-pool-snaps-readproxy} time: 00:30:01 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742215/ [5742216] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} tasks/insights} time: 00:10:01 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742216/ [5742217] rados/singleton/{all/pg-autoscaler-progress-off mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:16:30 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742217/ [5742218] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:17:50 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742218/ [5742219] rados/cephadm/orchestrator_cli/{2-node-mgr orchestrator_cli supported-random-distro$/{ubuntu_latest}} time: 00:09:33 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742219/ [5742220] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/classic start} time: 00:13:50 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742220/ [5742221] rados/cephadm/smoke-roleless/{distro/centos_8.0 start} time: 00:13:11 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742221/ [5742223] rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:18:36 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742223/ [5742224] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:08:14 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742224/ [5742225] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:09:12 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742225/ [5742226] rados/objectstore/{backends/fusestore supported-random-distro$/{ubuntu_latest}} time: 00:11:46 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742226/ [5742227] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:09:03 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742227/ [5742229] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:29:26 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742229/ [5742230] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python} time: 00:26:50 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742230/ [5742231] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/erasure-code} time: 00:44:22 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742231/ [5742232] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:11:30 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742232/ [5742233] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:29:55 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742233/ [5742234] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados tasks/mon_recovery validater/lockdep} time: 00:12:59 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742234/ [5742235] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:23:28 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742235/ [5742236] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} time: 00:37:54 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742236/ [5742237] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} tasks/readwrite} time: 00:12:54 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742237/ [5742238] rados/singleton/{all/pg-removal-interruption mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} time: 00:09:13 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742238/ [5742239] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache-snaps-balanced} time: 00:24:11 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742239/ [5742240] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:12:52 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742240/ [5742241] rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} tasks/mon_clock_with_skews} time: 00:08:15 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742241/ [5742242] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:30:37 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742242/ [5742243] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/connectivity start} time: 00:12:51 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742243/ [5742244] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_mon_workunits} time: 00:31:41 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742244/ [5742245] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/classic start} time: 00:15:01 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742245/ [5742246] rados/singleton/{all/radostool mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:18:29 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742246/ [5742247] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-snappy supported-random-distro$/{rhel_8} tasks/dashboard} time: 01:08:30 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742247/ [5742248] rados/cephadm/smoke-roleless/{distro/centos_latest start} time: 00:12:20 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742248/ [5742249] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/cache-snaps} time: 00:19:56 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742249/ [5742250] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:07:34 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742250/ [5742251] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:15:44 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742251/ [5742252] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} tasks/module_selftest} time: 00:24:53 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742252/ [5742253] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/test_rbd_api} time: 00:14:35 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742253/ [5742255] rados/singleton/{all/random-eio mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8}} time: 00:23:14 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742255/ [5742256] rados/standalone/{mon_election/connectivity supported-random-distro$/{rhel_8} workloads/mgr} time: 00:19:09 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742256/ [5742257] rados/cephadm/with-work/{distro/ubuntu_18.04_podman fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:38:35 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742257/ [5742258] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache} time: 00:12:46 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742258/ [5742259] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:09:14 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742259/ [5742260] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/connectivity start} time: 00:12:00 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742260/ [5742262] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/repair_test} time: 00:13:22 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742262/ [5742263] rados/singleton/{all/rebuild-mondb mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} time: 00:16:09 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742263/ [5742264] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} tasks/dashboard} time: 01:10:01 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742264/ [5742265] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{ubuntu_latest}} time: 00:11:46 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742265/ [5742266] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:38 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742266/ [5742267] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} time: 00:20:28 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742267/ [5742268] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/cache-snaps} time: 00:28:56 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742268/ [5742269] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:05:41 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742269/ [5742270] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/pool-snaps-few-objects} time: 00:35:23 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742270/ [5742271] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:10:57 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742271/ [5742273] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:23:24 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742273/ [5742274] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:22:06 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742274/ [5742275] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/classic start} time: 00:11:46 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742275/ [5742276] rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:08:10 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742276/ [5742277] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:26:53 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742277/ [5742278] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/snaps-few-objects} time: 00:32:04 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742278/ [5742279] rados/singleton/{all/recovery-preemption mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:19:19 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742279/ [5742280] rados/cephadm/smoke-roleless/{distro/rhel_8.0 start} time: 00:14:20 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742280/ [5742281] rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_python} time: 00:29:06 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742281/ [5742282] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:17:44 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742282/ [5742283] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:22:56 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742283/ [5742285] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/progress} time: 00:18:52 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742285/ [5742286] rados/standalone/{mon_election/classic supported-random-distro$/{centos_8} workloads/misc} time: 00:35:07 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742286/ [5742287] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/connectivity start} time: 00:12:15 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742287/ [5742288] rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8}} time: 00:08:55 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742288/ [5742289] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} tasks/dashboard} time: 01:10:51 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742289/ [5742290] rados/cephadm/workunits/{distro/centos_latest mon_election/classic task/test_cephadm_repos} time: 00:13:16 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742290/ [5742291] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:12:44 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742291/ [5742292] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/few rados thrashers/careful thrashosds-health workloads/radosbench} time: 01:45:28 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742292/ [5742294] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/sample_fio} time: 00:12:23 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742294/ [5742295] rados/cephadm/smoke/{distro/ubuntu_latest fixed-2 mon_election/classic start} time: 00:11:38 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742295/ [5742296] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} tasks/scrub_test} time: 00:12:48 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742296/ [5742297] rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} time: 00:14:10 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742297/ [5742298] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} time: 00:45:41 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742298/ [5742299] rados/cephadm/smoke-roleless/{distro/rhel_latest start} time: 00:14:29 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742299/ [5742300] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:42 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742300/ [5742301] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:10:36 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742301/ [5742302] rados/cephadm/with-work/{distro/centos_latest fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} time: 00:39:29 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742302/ [5742303] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/radosbench} time: 01:48:42 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742303/ [5742304] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/luminous-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/rbd_cls} time: 00:18:49 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742304/ [5742305] rados/singleton/{all/test_envlibrados_for_rocksdb mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} time: 00:19:53 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742305/ [5742306] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:26:50 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742306/ [5742307] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/lockdep} time: 00:37:16 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742307/ [5742308] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:06:07 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742308/ [5742309] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{ubuntu_latest}} time: 00:33:10 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742309/ [5742310] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:26:13 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742310/ [5742311] rados/standalone/{mon_election/connectivity supported-random-distro$/{ubuntu_latest} workloads/mon} time: 00:49:08 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742311/ [5742312] rados/cephadm/smoke/{distro/centos_8.0 fixed-2 mon_election/connectivity start} time: 00:13:34 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742312/ [5742313] rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews} time: 00:08:08 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742313/ [5742314] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:30:18 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742314/ [5742315] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/sync-many workloads/snaps-few-objects} time: 00:27:09 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742315/ [5742316] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:17:10 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742316/ [5742317] rados/cephadm/upgrade/{1-start 2-repo_digest/repo_digest 3-start-upgrade 4-wait distro$/{ubuntu_18.04_podman} fixed-2 mon_election/classic} time: 00:18:51 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742317/ [5742318] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/redirect} time: 00:11:54 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742318/ [5742319] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:08:22 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742319/ [5742320] rados/singleton/{all/thrash-backfill-full mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8}} time: 01:04:47 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742320/ [5742321] rados/cephadm/workunits/{distro/ubuntu_18.04_podman mon_election/connectivity task/test_orch_cli} time: 00:16:34 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742321/ [5742322] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8} tasks/prometheus} time: 00:11:55 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742322/ [5742323] rados/cephadm/smoke/{distro/centos_latest fixed-2 mon_election/classic start} time: 00:13:25 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742323/ [5742324] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:14:08 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742324/ [5742325] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04 start} time: 00:10:43 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742325/ [5742326] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} tasks/rados_api_tests} time: 00:33:08 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742326/ [5742327] rados/singleton/{all/thrash-eio mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} time: 00:37:29 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742327/ [5742328] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/redirect_promote_tests} time: 00:13:38 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742328/ [5742329] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:06:16 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742329/ [5742330] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/luminous backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/snaps-few-objects} time: 00:36:10 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742330/ [5742331] rados/cephadm/with-work/{distro/rhel_8.0 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python} time: 00:30:59 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742331/ [5742332] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:31:10 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742332/ [5742333] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:41 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742333/ [5742334] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:22:07 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742334/ [5742335] rados/standalone/{mon_election/classic supported-random-distro$/{ubuntu_latest} workloads/osd} time: 03:16:28 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742335/ [5742336] rados/cephadm/smoke/{distro/rhel_8.0 fixed-2 mon_election/connectivity start} time: 00:14:40 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742336/ [5742337] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}} time: 00:28:57 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742337/ [5742338] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/redirect_set_object} time: 00:10:37 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742338/ [5742339] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/cosbench_64K_read_write} time: 00:18:15 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742339/ [5742340] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/test_rbd_api} time: 00:17:48 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742340/ [5742341] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:08:09 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742341/ [5742342] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados tasks/rados_cls_all validater/valgrind} time: 02:21:24 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742342/ [5742343] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:27:08 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742343/ [5742344] rados/cephadm/smoke/{distro/rhel_latest fixed-2 mon_election/classic start} time: 00:12:25 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742344/ [5742345] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{centos_8}} time: 00:18:59 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742345/ [5742347] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} tasks/dashboard} time: 01:05:52 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742347/ [5742348] rados/objectstore/{backends/objectstore supported-random-distro$/{ubuntu_latest}} time: 05:15:26 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742348/ [5742349] rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} tasks/mon_clock_with_skews} time: 00:08:41 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742349/ [5742350] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-radosbench} time: 02:19:23 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742350/ [5742351] rados/cephadm/smoke-roleless/{distro/ubuntu_18.04_podman start} time: 00:17:59 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742351/ [5742352] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/sync workloads/pool-create-delete} time: 00:17:56 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742352/ [5742353] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/set-chunks-read} time: 00:12:16 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742353/ [5742354] rados/cephadm/workunits/{distro/centos_latest mon_election/connectivity task/test_adoption} time: 00:15:42 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742354/ [5742355] rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-snappy supported-random-distro$/{rhel_8} tasks/workunits} time: 00:16:51 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742355/ [5742357] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:12:29 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742357/ [5742358] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/rados_cls_all} time: 00:23:20 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742358/ [5742359] rados/singleton/{all/watch-notify-same-primary mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:09:41 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742359/ [5742360] rados/cephadm/with-work/{distro/rhel_latest fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} time: 00:39:46 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742360/ [5742362] rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/scrub} time: 01:46:51 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742362/ [5742363] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/cosbench_64K_write} time: 00:25:51 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742363/ [5742364] rados/cephadm/smoke/{distro/ubuntu_18.04 fixed-2 mon_election/connectivity start} time: 00:11:23 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742364/ [5742365] rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/cache-snaps} time: 00:25:33 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742365/ [5742366] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:29:27 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742366/ [5742367] rados/singleton/{all/admin-socket mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:04:49 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742367/ [5742368] rados/dashboard/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/filestore-xfs supported-random-distro$/{centos_8} tasks/dashboard} time: 01:07:48 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742368/ [5742369] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:12:44 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742369/ [5742370] rados/cephadm/smoke/{distro/ubuntu_18.04_podman fixed-2 mon_election/classic start} time: 00:12:35 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742370/ [5742371] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/small-objects-localized} time: 00:23:40 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742371/ [5742372] rados/cephadm/smoke-roleless/{distro/ubuntu_latest start} time: 00:10:45 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742372/ [5742373] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{centos_8}} time: 00:16:19 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742373/ [5742374] rados/singleton/{all/deduptool mon_election/connectivity msgr-failures/many msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8}} time: 00:16:33 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742374/ [5742375] rados/cephadm/with-work/{distro/ubuntu_18.04 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} time: 00:26:55 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742375/ [5742376] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:31:33 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742376/ [5742377] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados tasks/mon_recovery validater/lockdep} time: 00:12:47 info: http://pulpito.ceph.com/teuthology-2020-12-28_07:01:00-rados-master-distro-basic-smithi/5742377/