2023-07-07T05:32:21.613 INFO:root:teuthology version: 0.0.1.dev140+gb53f236
2023-07-07T05:32:21.614 INFO:teuthology.results:seed: '3194'
2023-07-07T05:33:12.917 INFO:root:teuthology version: 0.0.1.dev140+gb53f236
2023-07-07T05:33:12.920 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-07-07T17:34:01.772 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2023-07-07T17:34:03.343 INFO:scrape:Found 20 jobs
2023-07-07T17:34:03.511 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.0.0-4717-g6e5383fd (6e5383fdd6d402108195ddbbefb66fca2cebcf96) reef (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7efc469a6ed4]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x61b) [0x55e7f1e48535]

2023-07-07T17:34:03.513 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:01:02.676 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json
, 2023-07-07T06:01:02.804+0000 7f22ab646700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-07-07T06:01:02.810+0000 7f22ab646700 -1 WARNING: the following dangerous and experimental features are enabled: crimson

2023-07-07T17:34:03.514 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:01:11.727 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now
, 2023-07-07T06:01:11.727 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json
, 2023-07-07T06:01:11.853+0000 7f8496fe9700 -1 WARNING: the following dangerous and experimental features are enabled: crimson

2023-07-07T17:34:03.516 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:01:20.787 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now
, 2023-07-07T06:01:20.788 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json
, 2023-07-07T06:01:20.914+0000 7fe12ceba700 -1 WARNING: the following dangerous and experimental features are enabled: crimson

2023-07-07T17:34:03.524 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:01:29.826 INFO:teuthology.orchestra.run.smithi027.stdout:{"fsid":"421db855-4d57-41f4-bef0-9c6557345e90","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 92/704 objects degraded (13.068%), 4 pgs degraded","count":4},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":92,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":21,"num_osds":4,"num_up_osds":3,"osd_up_since":1688709658,"num_in_osds":4,"osd_in_since":1688709599,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+undersized","count":9},{"state_name":"active+undersized+degraded","count":4}],"num_pgs":25,"num_pools":3,"num_objects":352,"data_bytes":1022632651,"bytes_used":1450606592,"bytes_avail":428046123008,"bytes_total":429496729600,"degraded_objects":92,"degraded_total":704,"degraded_ratio":0.13068181818181818},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2023-07-07T06:01:05.781050+0000","services":{}},"progress_events":{}}
, 2023-07-07T06:01:29.840 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now
, 2023-07-07T06:01:29.841 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json

2023-07-07T17:34:03.551 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:01:38.874 INFO:teuthology.orchestra.run.smithi027.stdout:
, 2023-07-07T06:01:38.874 INFO:teuthology.orchestra.run.smithi027.stdout:{"fsid":"421db855-4d57-41f4-bef0-9c6557345e90","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 92/704 objects degraded (13.068%), 4 pgs degraded","count":4},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":101,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":21,"num_osds":4,"num_up_osds":3,"osd_up_since":1688709658,"num_in_osds":4,"osd_in_since":1688709599,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+undersized","count":9},{"state_name":"active+undersized+degraded","count":4}],"num_pgs":25,"num_pools":3,"num_objects":352,"data_bytes":1022632651,"bytes_used":1450606592,"bytes_avail":428046123008,"bytes_total":429496729600,"degraded_objects":92,"degraded_total":704,"degraded_ratio":0.13068181818181818},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2023-07-07T06:01:05.781050+0000","services":{}},"progress_events":{}}
, 2023-07-07T06:01:38.888 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now

2023-07-07T17:34:03.552 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:01:47.925 INFO:teuthology.orchestra.run.smithi027.stdout:
, 2023-07-07T06:01:47.925 INFO:teuthology.orchestra.run.smithi027.stdout:{"fsid":"421db855-4d57-41f4-bef0-9c6557345e90","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 92/704 objects degraded (13.068%), 4 pgs degraded","count":4},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":110,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":21,"num_osds":4,"num_up_osds":3,"osd_up_since":1688709658,"num_in_osds":4,"osd_in_since":1688709599,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+undersized","count":9},{"state_name":"active+undersized+degraded","count":4}],"num_pgs":25,"num_pools":3,"num_objects":352,"data_bytes":1022632651,"bytes_used":1450606592,"bytes_avail":428046123008,"bytes_total":429496729600,"degraded_objects":92,"degraded_total":704,"degraded_ratio":0.13068181818181818},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2023-07-07T06:01:05.781050+0000","services":{}},"progress_events":{}}
, 2023-07-07T06:01:47.940 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now

2023-07-07T17:34:03.554 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:01:56.693+0000 7f654f336700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-07-07T06:01:56.982 INFO:teuthology.orchestra.run.smithi027.stdout:
, 2023-07-07T06:01:56.982 INFO:teuthology.orchestra.run.smithi027.stdout:{"fsid":"421db855-4d57-41f4-bef0-9c6557345e90","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 92/704 objects degraded (13.068%), 4 pgs degraded","count":4},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":119,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":21,"num_osds":4,"num_up_osds":3,"osd_up_since":1688709658,"num_in_osds":4,"osd_in_since":1688709599,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+undersized","count":9},{"state_name":"active+undersized+degraded","count":4}],"num_pgs":25,"num_pools":3,"num_objects":352,"data_bytes":1022632651,"bytes_used":1450606592,"bytes_avail":428046123008,"bytes_total":429496729600,"degraded_objects":92,"degraded_total":704,"degraded_ratio":0.13068181818181818},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2023-07-07T06:01:05.781050+0000","services":{}},"progress_events":{}}

2023-07-07T17:34:03.555 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:02:05.737+0000 7fdc4ab1e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-07-07T06:02:05.928 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~88s
, 2023-07-07T06:02:06.025 INFO:teuthology.orchestra.run.smithi027.stdout:

2023-07-07T17:34:03.557 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:02:14.795+0000 7f4aa3027700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-07-07T06:02:14.800+0000 7f4aa3027700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-07-07T06:02:15.091 INFO:teuthology.orchestra.run.smithi027.stdout:

2023-07-07T17:34:03.558 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:02:23.724 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json
, 2023-07-07T06:02:23.850+0000 7fcfaaae6700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-07-07T06:02:23.856+0000 7fcfaaae6700 -1 WARNING: the following dangerous and experimental features are enabled: crimson

2023-07-07T17:34:03.560 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:02:32.767 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json
, 2023-07-07T06:02:32.893+0000 7f7590d47700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-07-07T06:02:32.899+0000 7f7590d47700 -1 WARNING: the following dangerous and experimental features are enabled: crimson

2023-07-07T17:34:03.561 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:02:41.818 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.462985+0000', 'last_change': '2023-07-07T06:00:59.462558+0000', 'last_active': '2023-07-07T06:00:59.462985+0000', 'last_peered': '2023-07-07T06:00:59.462985+0000', 'last_clean': '2023-07-07T06:00:25.415191+0000', 'last_became_active': '2023-07-07T06:00:59.462558+0000', 'last_became_peered': '2023-07-07T06:00:59.462558+0000', 'last_unstale': '2023-07-07T06:00:59.462985+0000', 'last_undegraded': '2023-07-07T06:00:59.462985+0000', 'last_fullsized': '2023-07-07T06:00:59.453964+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:02:41.819 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json
, 2023-07-07T06:02:41.944+0000 7fd8c412b700 -1 WARNING: the following dangerous and experimental features are enabled: crimson

2023-07-07T17:34:03.564 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:02:50.884 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean
, 2023-07-07T06:02:50.884 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.462985+0000', 'last_change': '2023-07-07T06:00:59.462558+0000', 'last_active': '2023-07-07T06:00:59.462985+0000', 'last_peered': '2023-07-07T06:00:59.462985+0000', 'last_clean': '2023-07-07T06:00:25.415191+0000', 'last_became_active': '2023-07-07T06:00:59.462558+0000', 'last_became_peered': '2023-07-07T06:00:59.462558+0000', 'last_unstale': '2023-07-07T06:00:59.462985+0000', 'last_undegraded': '2023-07-07T06:00:59.462985+0000', 'last_fullsized': '2023-07-07T06:00:59.453964+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:02:50.885 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json

2023-07-07T17:34:03.587 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:02:59.945 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean
, 2023-07-07T06:02:59.946 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.462985+0000', 'last_change': '2023-07-07T06:00:59.462558+0000', 'last_active': '2023-07-07T06:00:59.462985+0000', 'last_peered': '2023-07-07T06:00:59.462985+0000', 'last_clean': '2023-07-07T06:00:25.415191+0000', 'last_became_active': '2023-07-07T06:00:59.462558+0000', 'last_became_peered': '2023-07-07T06:00:59.462558+0000', 'last_unstale': '2023-07-07T06:00:59.462985+0000', 'last_undegraded': '2023-07-07T06:00:59.462985+0000', 'last_fullsized': '2023-07-07T06:00:59.453964+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:02:59.946 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json

2023-07-07T17:34:03.589 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:03:09.001 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.463127+0000', 'last_change': '2023-07-07T06:00:59.462748+0000', 'last_active': '2023-07-07T06:00:59.463127+0000', 'last_peered': '2023-07-07T06:00:59.463127+0000', 'last_clean': '2023-07-07T06:00:22.399960+0000', 'last_became_active': '2023-07-07T06:00:59.462748+0000', 'last_became_peered': '2023-07-07T06:00:59.462748+0000', 'last_unstale': '2023-07-07T06:00:59.463127+0000', 'last_undegraded': '2023-07-07T06:00:59.463127+0000', 'last_fullsized': '2023-07-07T06:00:59.455926+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []}
, 2023-07-07T06:03:09.002 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean
, 2023-07-07T06:03:09.002 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.462985+0000', 'last_change': '2023-07-07T06:00:59.462558+0000', 'last_active': '2023-07-07T06:00:59.462985+0000', 'last_peered': '2023-07-07T06:00:59.462985+0000', 'last_clean': '2023-07-07T06:00:25.415191+0000', 'last_became_active': '2023-07-07T06:00:59.462558+0000', 'last_became_peered': '2023-07-07T06:00:59.462558+0000', 'last_unstale': '2023-07-07T06:00:59.462985+0000', 'last_undegraded': '2023-07-07T06:00:59.462985+0000', 'last_fullsized': '2023-07-07T06:00:59.453964+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}

2023-07-07T17:34:03.590 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:03:18.048 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean
, 2023-07-07T06:03:18.049 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.463127+0000', 'last_change': '2023-07-07T06:00:59.462748+0000', 'last_active': '2023-07-07T06:00:59.463127+0000', 'last_peered': '2023-07-07T06:00:59.463127+0000', 'last_clean': '2023-07-07T06:00:22.399960+0000', 'last_became_active': '2023-07-07T06:00:59.462748+0000', 'last_became_peered': '2023-07-07T06:00:59.462748+0000', 'last_unstale': '2023-07-07T06:00:59.463127+0000', 'last_undegraded': '2023-07-07T06:00:59.463127+0000', 'last_fullsized': '2023-07-07T06:00:59.455926+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []}
, 2023-07-07T06:03:18.049 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean

2023-07-07T17:34:03.592 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:03:27.092 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean
, 2023-07-07T06:03:27.093 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.463127+0000', 'last_change': '2023-07-07T06:00:59.462748+0000', 'last_active': '2023-07-07T06:00:59.463127+0000', 'last_peered': '2023-07-07T06:00:59.463127+0000', 'last_clean': '2023-07-07T06:00:22.399960+0000', 'last_became_active': '2023-07-07T06:00:59.462748+0000', 'last_became_peered': '2023-07-07T06:00:59.462748+0000', 'last_unstale': '2023-07-07T06:00:59.463127+0000', 'last_undegraded': '2023-07-07T06:00:59.463127+0000', 'last_fullsized': '2023-07-07T06:00:59.455926+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []}
, 2023-07-07T06:03:27.093 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean

2023-07-07T17:34:03.593 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:03:36.152 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.463407+0000', 'last_change': '2023-07-07T06:00:59.462897+0000', 'last_active': '2023-07-07T06:00:59.463407+0000', 'last_peered': '2023-07-07T06:00:59.463407+0000', 'last_clean': '2023-07-07T06:00:08.305657+0000', 'last_became_active': '2023-07-07T06:00:59.462897+0000', 'last_became_peered': '2023-07-07T06:00:59.462897+0000', 'last_unstale': '2023-07-07T06:00:59.463407+0000', 'last_undegraded': '2023-07-07T06:00:59.463407+0000', 'last_fullsized': '2023-07-07T06:00:59.454340+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:03:36.152 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean
, 2023-07-07T06:03:36.152 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.463127+0000', 'last_change': '2023-07-07T06:00:59.462748+0000', 'last_active': '2023-07-07T06:00:59.463127+0000', 'last_peered': '2023-07-07T06:00:59.463127+0000', 'last_clean': '2023-07-07T06:00:22.399960+0000', 'last_became_active': '2023-07-07T06:00:59.462748+0000', 'last_became_peered': '2023-07-07T06:00:59.462748+0000', 'last_unstale': '2023-07-07T06:00:59.463127+0000', 'last_undegraded': '2023-07-07T06:00:59.463127+0000', 'last_fullsized': '2023-07-07T06:00:59.455926+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []}

2023-07-07T17:34:03.595 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:03:45.219 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean
, 2023-07-07T06:03:45.219 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.463407+0000', 'last_change': '2023-07-07T06:00:59.462897+0000', 'last_active': '2023-07-07T06:00:59.463407+0000', 'last_peered': '2023-07-07T06:00:59.463407+0000', 'last_clean': '2023-07-07T06:00:08.305657+0000', 'last_became_active': '2023-07-07T06:00:59.462897+0000', 'last_became_peered': '2023-07-07T06:00:59.462897+0000', 'last_unstale': '2023-07-07T06:00:59.463407+0000', 'last_undegraded': '2023-07-07T06:00:59.463407+0000', 'last_fullsized': '2023-07-07T06:00:59.454340+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:03:45.220 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean

2023-07-07T17:34:03.596 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:03:54.275 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean
, 2023-07-07T06:03:54.275 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.463407+0000', 'last_change': '2023-07-07T06:00:59.462897+0000', 'last_active': '2023-07-07T06:00:59.463407+0000', 'last_peered': '2023-07-07T06:00:59.463407+0000', 'last_clean': '2023-07-07T06:00:08.305657+0000', 'last_became_active': '2023-07-07T06:00:59.462897+0000', 'last_became_peered': '2023-07-07T06:00:59.462897+0000', 'last_unstale': '2023-07-07T06:00:59.463407+0000', 'last_undegraded': '2023-07-07T06:00:59.463407+0000', 'last_fullsized': '2023-07-07T06:00:59.454340+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:03:54.275 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean

2023-07-07T17:34:03.598 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:04:03.349 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2', 'version': "19'123", 'reported_seq': 192, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.480068+0000', 'last_change': '2023-07-07T06:00:59.470873+0000', 'last_active': '2023-07-07T06:00:59.480068+0000', 'last_peered': '2023-07-07T06:00:59.480068+0000', 'last_clean': '2023-07-07T06:00:35.066600+0000', 'last_became_active': '2023-07-07T06:00:59.470873+0000', 'last_became_peered': '2023-07-07T06:00:59.470873+0000', 'last_unstale': '2023-07-07T06:00:59.480068+0000', 'last_undegraded': '2023-07-07T06:00:59.462245+0000', 'last_fullsized': '2023-07-07T06:00:59.461717+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'objects_scrubbed': 0, 'log_size': 123, 'log_dups_size': 0, 'ondisk_log_size': 123, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 96893477, 'num_objects': 34, 'num_object_clones': 0, 'num_object_copies': 68, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 34, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 107, 'num_read_kb': 31202, 'num_write': 158, 'num_write_kb': 47074, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 34}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:04:03.349 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean
, 2023-07-07T06:04:03.349 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.463407+0000', 'last_change': '2023-07-07T06:00:59.462897+0000', 'last_active': '2023-07-07T06:00:59.463407+0000', 'last_peered': '2023-07-07T06:00:59.463407+0000', 'last_clean': '2023-07-07T06:00:08.305657+0000', 'last_became_active': '2023-07-07T06:00:59.462897+0000', 'last_became_peered': '2023-07-07T06:00:59.462897+0000', 'last_unstale': '2023-07-07T06:00:59.463407+0000', 'last_undegraded': '2023-07-07T06:00:59.463407+0000', 'last_fullsized': '2023-07-07T06:00:59.454340+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}

2023-07-07T17:34:03.660 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:04:12.418 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2', 'version': "19'123", 'reported_seq': 192, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.480068+0000', 'last_change': '2023-07-07T06:00:59.470873+0000', 'last_active': '2023-07-07T06:00:59.480068+0000', 'last_peered': '2023-07-07T06:00:59.480068+0000', 'last_clean': '2023-07-07T06:00:35.066600+0000', 'last_became_active': '2023-07-07T06:00:59.470873+0000', 'last_became_peered': '2023-07-07T06:00:59.470873+0000', 'last_unstale': '2023-07-07T06:00:59.480068+0000', 'last_undegraded': '2023-07-07T06:00:59.462245+0000', 'last_fullsized': '2023-07-07T06:00:59.461717+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'objects_scrubbed': 0, 'log_size': 123, 'log_dups_size': 0, 'ondisk_log_size': 123, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 96893477, 'num_objects': 34, 'num_object_clones': 0, 'num_object_copies': 68, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 34, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 107, 'num_read_kb': 31202, 'num_write': 158, 'num_write_kb': 47074, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 34}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:04:12.419 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean
, 2023-07-07T06:04:12.419 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.463407+0000', 'last_change': '2023-07-07T06:00:59.462897+0000', 'last_active': '2023-07-07T06:00:59.463407+0000', 'last_peered': '2023-07-07T06:00:59.463407+0000', 'last_clean': '2023-07-07T06:00:08.305657+0000', 'last_became_active': '2023-07-07T06:00:59.462897+0000', 'last_became_peered': '2023-07-07T06:00:59.462897+0000', 'last_unstale': '2023-07-07T06:00:59.463407+0000', 'last_undegraded': '2023-07-07T06:00:59.463407+0000', 'last_fullsized': '2023-07-07T06:00:59.454340+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}

2023-07-07T17:34:03.662 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:04:21.470 INFO:tasks.ceph.ceph_manager.ceph:PG 3.2 is not active+clean
, 2023-07-07T06:04:21.471 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2', 'version': "19'123", 'reported_seq': 192, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.480068+0000', 'last_change': '2023-07-07T06:00:59.470873+0000', 'last_active': '2023-07-07T06:00:59.480068+0000', 'last_peered': '2023-07-07T06:00:59.480068+0000', 'last_clean': '2023-07-07T06:00:35.066600+0000', 'last_became_active': '2023-07-07T06:00:59.470873+0000', 'last_became_peered': '2023-07-07T06:00:59.470873+0000', 'last_unstale': '2023-07-07T06:00:59.480068+0000', 'last_undegraded': '2023-07-07T06:00:59.462245+0000', 'last_fullsized': '2023-07-07T06:00:59.461717+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'objects_scrubbed': 0, 'log_size': 123, 'log_dups_size': 0, 'ondisk_log_size': 123, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 96893477, 'num_objects': 34, 'num_object_clones': 0, 'num_object_copies': 68, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 34, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 107, 'num_read_kb': 31202, 'num_write': 158, 'num_write_kb': 47074, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 34}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:04:21.471 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean

2023-07-07T17:34:03.663 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:04:30.516 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "14'97", 'reported_seq': 155, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.469971+0000', 'last_change': '2023-07-07T06:00:59.469575+0000', 'last_active': '2023-07-07T06:00:59.469971+0000', 'last_peered': '2023-07-07T06:00:59.469971+0000', 'last_clean': '2023-07-07T06:00:23.369029+0000', 'last_became_active': '2023-07-07T06:00:59.469575+0000', 'last_became_peered': '2023-07-07T06:00:59.469575+0000', 'last_unstale': '2023-07-07T06:00:59.469971+0000', 'last_undegraded': '2023-07-07T06:00:59.465049+0000', 'last_fullsized': '2023-07-07T06:00:59.464546+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []}
, 2023-07-07T06:04:30.516 INFO:tasks.ceph.ceph_manager.ceph:PG 3.2 is not active+clean
, 2023-07-07T06:04:30.517 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2', 'version': "19'123", 'reported_seq': 192, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.480068+0000', 'last_change': '2023-07-07T06:00:59.470873+0000', 'last_active': '2023-07-07T06:00:59.480068+0000', 'last_peered': '2023-07-07T06:00:59.480068+0000', 'last_clean': '2023-07-07T06:00:35.066600+0000', 'last_became_active': '2023-07-07T06:00:59.470873+0000', 'last_became_peered': '2023-07-07T06:00:59.470873+0000', 'last_unstale': '2023-07-07T06:00:59.480068+0000', 'last_undegraded': '2023-07-07T06:00:59.462245+0000', 'last_fullsized': '2023-07-07T06:00:59.461717+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'objects_scrubbed': 0, 'log_size': 123, 'log_dups_size': 0, 'ondisk_log_size': 123, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 96893477, 'num_objects': 34, 'num_object_clones': 0, 'num_object_copies': 68, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 34, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 107, 'num_read_kb': 31202, 'num_write': 158, 'num_write_kb': 47074, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 34}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}

2023-07-07T17:34:03.665 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:04:39.576 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "14'97", 'reported_seq': 155, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.469971+0000', 'last_change': '2023-07-07T06:00:59.469575+0000', 'last_active': '2023-07-07T06:00:59.469971+0000', 'last_peered': '2023-07-07T06:00:59.469971+0000', 'last_clean': '2023-07-07T06:00:23.369029+0000', 'last_became_active': '2023-07-07T06:00:59.469575+0000', 'last_became_peered': '2023-07-07T06:00:59.469575+0000', 'last_unstale': '2023-07-07T06:00:59.469971+0000', 'last_undegraded': '2023-07-07T06:00:59.465049+0000', 'last_fullsized': '2023-07-07T06:00:59.464546+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []}
, 2023-07-07T06:04:39.576 INFO:tasks.ceph.ceph_manager.ceph:PG 3.2 is not active+clean
, 2023-07-07T06:04:39.577 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2', 'version': "19'123", 'reported_seq': 192, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.480068+0000', 'last_change': '2023-07-07T06:00:59.470873+0000', 'last_active': '2023-07-07T06:00:59.480068+0000', 'last_peered': '2023-07-07T06:00:59.480068+0000', 'last_clean': '2023-07-07T06:00:35.066600+0000', 'last_became_active': '2023-07-07T06:00:59.470873+0000', 'last_became_peered': '2023-07-07T06:00:59.470873+0000', 'last_unstale': '2023-07-07T06:00:59.480068+0000', 'last_undegraded': '2023-07-07T06:00:59.462245+0000', 'last_fullsized': '2023-07-07T06:00:59.461717+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'objects_scrubbed': 0, 'log_size': 123, 'log_dups_size': 0, 'ondisk_log_size': 123, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 96893477, 'num_objects': 34, 'num_object_clones': 0, 'num_object_copies': 68, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 34, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 107, 'num_read_kb': 31202, 'num_write': 158, 'num_write_kb': 47074, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 34}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}

2023-07-07T17:34:03.666 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:04:48.624 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean
, 2023-07-07T06:04:48.624 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "14'97", 'reported_seq': 155, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.469971+0000', 'last_change': '2023-07-07T06:00:59.469575+0000', 'last_active': '2023-07-07T06:00:59.469971+0000', 'last_peered': '2023-07-07T06:00:59.469971+0000', 'last_clean': '2023-07-07T06:00:23.369029+0000', 'last_became_active': '2023-07-07T06:00:59.469575+0000', 'last_became_peered': '2023-07-07T06:00:59.469575+0000', 'last_unstale': '2023-07-07T06:00:59.469971+0000', 'last_undegraded': '2023-07-07T06:00:59.465049+0000', 'last_fullsized': '2023-07-07T06:00:59.464546+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []}
, 2023-07-07T06:04:48.625 INFO:tasks.ceph.ceph_manager.ceph:PG 3.2 is not active+clean

2023-07-07T17:34:03.668 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:04:57.681 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.468569+0000', 'last_change': '2023-07-07T06:00:59.468122+0000', 'last_active': '2023-07-07T06:00:59.468569+0000', 'last_peered': '2023-07-07T06:00:59.468569+0000', 'last_clean': '2023-07-07T06:00:25.415275+0000', 'last_became_active': '2023-07-07T06:00:59.468122+0000', 'last_became_peered': '2023-07-07T06:00:59.468122+0000', 'last_unstale': '2023-07-07T06:00:59.468569+0000', 'last_undegraded': '2023-07-07T06:00:59.468569+0000', 'last_fullsized': '2023-07-07T06:00:59.457597+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:04:57.682 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean
, 2023-07-07T06:04:57.682 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "14'97", 'reported_seq': 155, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.469971+0000', 'last_change': '2023-07-07T06:00:59.469575+0000', 'last_active': '2023-07-07T06:00:59.469971+0000', 'last_peered': '2023-07-07T06:00:59.469971+0000', 'last_clean': '2023-07-07T06:00:23.369029+0000', 'last_became_active': '2023-07-07T06:00:59.469575+0000', 'last_became_peered': '2023-07-07T06:00:59.469575+0000', 'last_unstale': '2023-07-07T06:00:59.469971+0000', 'last_undegraded': '2023-07-07T06:00:59.465049+0000', 'last_fullsized': '2023-07-07T06:00:59.464546+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []}

2023-07-07T17:34:03.669 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:05:06.737 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.468569+0000', 'last_change': '2023-07-07T06:00:59.468122+0000', 'last_active': '2023-07-07T06:00:59.468569+0000', 'last_peered': '2023-07-07T06:00:59.468569+0000', 'last_clean': '2023-07-07T06:00:25.415275+0000', 'last_became_active': '2023-07-07T06:00:59.468122+0000', 'last_became_peered': '2023-07-07T06:00:59.468122+0000', 'last_unstale': '2023-07-07T06:00:59.468569+0000', 'last_undegraded': '2023-07-07T06:00:59.468569+0000', 'last_fullsized': '2023-07-07T06:00:59.457597+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:05:06.738 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean
, 2023-07-07T06:05:06.738 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "14'97", 'reported_seq': 155, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.469971+0000', 'last_change': '2023-07-07T06:00:59.469575+0000', 'last_active': '2023-07-07T06:00:59.469971+0000', 'last_peered': '2023-07-07T06:00:59.469971+0000', 'last_clean': '2023-07-07T06:00:23.369029+0000', 'last_became_active': '2023-07-07T06:00:59.469575+0000', 'last_became_peered': '2023-07-07T06:00:59.469575+0000', 'last_unstale': '2023-07-07T06:00:59.469971+0000', 'last_undegraded': '2023-07-07T06:00:59.465049+0000', 'last_fullsized': '2023-07-07T06:00:59.464546+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []}

2023-07-07T17:34:03.671 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:05:15.804 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean
, 2023-07-07T06:05:15.805 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.468569+0000', 'last_change': '2023-07-07T06:00:59.468122+0000', 'last_active': '2023-07-07T06:00:59.468569+0000', 'last_peered': '2023-07-07T06:00:59.468569+0000', 'last_clean': '2023-07-07T06:00:25.415275+0000', 'last_became_active': '2023-07-07T06:00:59.468122+0000', 'last_became_peered': '2023-07-07T06:00:59.468122+0000', 'last_unstale': '2023-07-07T06:00:59.468569+0000', 'last_undegraded': '2023-07-07T06:00:59.468569+0000', 'last_fullsized': '2023-07-07T06:00:59.457597+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:05:15.805 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean

2023-07-07T17:34:03.672 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:05:24.890 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.3', 'version': "21'107", 'reported_seq': 156, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.478255+0000', 'last_change': '2023-07-07T06:00:59.464604+0000', 'last_active': '2023-07-07T06:00:59.478255+0000', 'last_peered': '2023-07-07T06:00:59.478255+0000', 'last_clean': '2023-07-07T06:00:33.546473+0000', 'last_became_active': '2023-07-07T06:00:59.464604+0000', 'last_became_peered': '2023-07-07T06:00:59.464604+0000', 'last_unstale': '2023-07-07T06:00:59.478255+0000', 'last_undegraded': '2023-07-07T06:00:59.458856+0000', 'last_fullsized': '2023-07-07T06:00:59.458231+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'objects_scrubbed': 0, 'log_size': 107, 'log_dups_size': 0, 'ondisk_log_size': 107, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 82623806, 'num_objects': 29, 'num_object_clones': 0, 'num_object_copies': 58, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 29, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 61, 'num_read_kb': 31, 'num_write': 138, 'num_write_kb': 39292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 29}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []}
, 2023-07-07T06:05:24.891 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean
, 2023-07-07T06:05:24.891 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.468569+0000', 'last_change': '2023-07-07T06:00:59.468122+0000', 'last_active': '2023-07-07T06:00:59.468569+0000', 'last_peered': '2023-07-07T06:00:59.468569+0000', 'last_clean': '2023-07-07T06:00:25.415275+0000', 'last_became_active': '2023-07-07T06:00:59.468122+0000', 'last_became_peered': '2023-07-07T06:00:59.468122+0000', 'last_unstale': '2023-07-07T06:00:59.468569+0000', 'last_undegraded': '2023-07-07T06:00:59.468569+0000', 'last_fullsized': '2023-07-07T06:00:59.457597+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}

2023-07-07T17:34:03.673 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:05:33.966 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.3', 'version': "21'107", 'reported_seq': 156, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.478255+0000', 'last_change': '2023-07-07T06:00:59.464604+0000', 'last_active': '2023-07-07T06:00:59.478255+0000', 'last_peered': '2023-07-07T06:00:59.478255+0000', 'last_clean': '2023-07-07T06:00:33.546473+0000', 'last_became_active': '2023-07-07T06:00:59.464604+0000', 'last_became_peered': '2023-07-07T06:00:59.464604+0000', 'last_unstale': '2023-07-07T06:00:59.478255+0000', 'last_undegraded': '2023-07-07T06:00:59.458856+0000', 'last_fullsized': '2023-07-07T06:00:59.458231+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'objects_scrubbed': 0, 'log_size': 107, 'log_dups_size': 0, 'ondisk_log_size': 107, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 82623806, 'num_objects': 29, 'num_object_clones': 0, 'num_object_copies': 58, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 29, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 61, 'num_read_kb': 31, 'num_write': 138, 'num_write_kb': 39292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 29}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []}
, 2023-07-07T06:05:33.967 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean
, 2023-07-07T06:05:33.967 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.468569+0000', 'last_change': '2023-07-07T06:00:59.468122+0000', 'last_active': '2023-07-07T06:00:59.468569+0000', 'last_peered': '2023-07-07T06:00:59.468569+0000', 'last_clean': '2023-07-07T06:00:25.415275+0000', 'last_became_active': '2023-07-07T06:00:59.468122+0000', 'last_became_peered': '2023-07-07T06:00:59.468122+0000', 'last_unstale': '2023-07-07T06:00:59.468569+0000', 'last_undegraded': '2023-07-07T06:00:59.468569+0000', 'last_fullsized': '2023-07-07T06:00:59.457597+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}

2023-07-07T17:34:03.674 WARNING:scrape:Ignoring malparsed backtrace:   File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception
,   File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise
,     raise value.with_traceback(tb)

2023-07-07T17:34:03.675 WARNING:scrape:Ignoring malparsed backtrace: Traceback (most recent call last):
,   File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/daemon/state.py", line 139, in stop
,     run.wait([self.proc], timeout=timeout)

2023-07-07T17:34:03.675 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 479, in wait
,     proc.wait()
,   File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait

2023-07-07T17:34:03.676 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait
,     self._raise_for_status()
,   File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status

2023-07-07T17:34:03.676 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.115332.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116271.log
,  49.6% -- replaced with /var/log/ceph/ceph-client.admin.115332.log.gz
, /var/log/ceph/ceph-client.admin.121237.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114695.log

2023-07-07T17:34:03.677 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.123891.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120395.log
,  49.2% -- replaced with /var/log/ceph/ceph-client.admin.123891.log.gz
, /var/log/ceph/ceph-client.admin.124136.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121678.log

2023-07-07T17:34:03.677 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.123695.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120698.log
,  49.6% -- replaced with /var/log/ceph/ceph-client.admin.123695.log.gz
, /var/log/ceph/ceph-client.admin.117300.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113421.log

2023-07-07T17:34:03.678 WARNING:scrape:Ignoring malparsed backtrace:  50.0% -- replaced with /var/log/ceph/ceph-client.admin.114303.log.gz
, gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122217.log
, /var/log/ceph/ceph-client.admin.123001.log:	 49.6% -- replaced with /var/log/ceph/ceph-client.admin.123001.log.gz/var/log/ceph/ceph-client.admin.111201.log:

2023-07-07T17:34:03.678 WARNING:scrape:Ignoring malparsed backtrace:  49.6% -- replaced with /var/log/ceph/ceph-client.admin.117643.log.gz
, /var/log/ceph/ceph-client.admin.122119.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123940.log
,  49.6% -- replaced with /var/log/ceph/ceph-client.admin.122119.log.gz

2023-07-07T17:34:03.679 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.121531.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118288.log
,  50.0% -- replaced with /var/log/ceph/ceph-client.admin.121531.log.gz
, /var/log/ceph/ceph-client.admin.118974.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115781.log

2023-07-07T17:34:03.679 WARNING:scrape:Ignoring malparsed backtrace: Traceback (most recent call last):
,   File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run_tasks.py", line 192, in run_tasks
,     suppress = manager.__exit__(*exc_info)

2023-07-07T17:34:03.679 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:07:45.041 DEBUG:teuthology.orchestra.run.smithi027:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz
, 2023-07-07T06:07:45.063 DEBUG:teuthology.orchestra.run.smithi167:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz
, 2023-07-07T06:07:45.145 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo

2023-07-07T17:34:03.702 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.0.0-4717-g6e5383fd (6e5383fdd6d402108195ddbbefb66fca2cebcf96) reef (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7efc469a6ed4]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x61b) [0x55e7f1e48535]

2023-07-07T17:34:03.704 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:01:02.676 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json
, 2023-07-07T06:01:02.804+0000 7f22ab646700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-07-07T06:01:02.810+0000 7f22ab646700 -1 WARNING: the following dangerous and experimental features are enabled: crimson

2023-07-07T17:34:03.706 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:01:11.727 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now
, 2023-07-07T06:01:11.727 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json
, 2023-07-07T06:01:11.853+0000 7f8496fe9700 -1 WARNING: the following dangerous and experimental features are enabled: crimson

2023-07-07T17:34:03.707 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:01:20.787 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now
, 2023-07-07T06:01:20.788 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json
, 2023-07-07T06:01:20.914+0000 7fe12ceba700 -1 WARNING: the following dangerous and experimental features are enabled: crimson

2023-07-07T17:34:03.708 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:01:29.826 INFO:teuthology.orchestra.run.smithi027.stdout:{"fsid":"421db855-4d57-41f4-bef0-9c6557345e90","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 92/704 objects degraded (13.068%), 4 pgs degraded","count":4},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":92,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":21,"num_osds":4,"num_up_osds":3,"osd_up_since":1688709658,"num_in_osds":4,"osd_in_since":1688709599,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+undersized","count":9},{"state_name":"active+undersized+degraded","count":4}],"num_pgs":25,"num_pools":3,"num_objects":352,"data_bytes":1022632651,"bytes_used":1450606592,"bytes_avail":428046123008,"bytes_total":429496729600,"degraded_objects":92,"degraded_total":704,"degraded_ratio":0.13068181818181818},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2023-07-07T06:01:05.781050+0000","services":{}},"progress_events":{}}
, 2023-07-07T06:01:29.840 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now
, 2023-07-07T06:01:29.841 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json

2023-07-07T17:34:03.710 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:01:38.874 INFO:teuthology.orchestra.run.smithi027.stdout:
, 2023-07-07T06:01:38.874 INFO:teuthology.orchestra.run.smithi027.stdout:{"fsid":"421db855-4d57-41f4-bef0-9c6557345e90","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 92/704 objects degraded (13.068%), 4 pgs degraded","count":4},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":101,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":21,"num_osds":4,"num_up_osds":3,"osd_up_since":1688709658,"num_in_osds":4,"osd_in_since":1688709599,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+undersized","count":9},{"state_name":"active+undersized+degraded","count":4}],"num_pgs":25,"num_pools":3,"num_objects":352,"data_bytes":1022632651,"bytes_used":1450606592,"bytes_avail":428046123008,"bytes_total":429496729600,"degraded_objects":92,"degraded_total":704,"degraded_ratio":0.13068181818181818},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2023-07-07T06:01:05.781050+0000","services":{}},"progress_events":{}}
, 2023-07-07T06:01:38.888 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now

2023-07-07T17:34:03.711 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:01:47.925 INFO:teuthology.orchestra.run.smithi027.stdout:
, 2023-07-07T06:01:47.925 INFO:teuthology.orchestra.run.smithi027.stdout:{"fsid":"421db855-4d57-41f4-bef0-9c6557345e90","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 92/704 objects degraded (13.068%), 4 pgs degraded","count":4},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":110,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":21,"num_osds":4,"num_up_osds":3,"osd_up_since":1688709658,"num_in_osds":4,"osd_in_since":1688709599,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+undersized","count":9},{"state_name":"active+undersized+degraded","count":4}],"num_pgs":25,"num_pools":3,"num_objects":352,"data_bytes":1022632651,"bytes_used":1450606592,"bytes_avail":428046123008,"bytes_total":429496729600,"degraded_objects":92,"degraded_total":704,"degraded_ratio":0.13068181818181818},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2023-07-07T06:01:05.781050+0000","services":{}},"progress_events":{}}
, 2023-07-07T06:01:47.940 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now

2023-07-07T17:34:03.712 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:01:56.693+0000 7f654f336700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-07-07T06:01:56.982 INFO:teuthology.orchestra.run.smithi027.stdout:
, 2023-07-07T06:01:56.982 INFO:teuthology.orchestra.run.smithi027.stdout:{"fsid":"421db855-4d57-41f4-bef0-9c6557345e90","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 92/704 objects degraded (13.068%), 4 pgs degraded","count":4},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":119,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":21,"num_osds":4,"num_up_osds":3,"osd_up_since":1688709658,"num_in_osds":4,"osd_in_since":1688709599,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+undersized","count":9},{"state_name":"active+undersized+degraded","count":4}],"num_pgs":25,"num_pools":3,"num_objects":352,"data_bytes":1022632651,"bytes_used":1450606592,"bytes_avail":428046123008,"bytes_total":429496729600,"degraded_objects":92,"degraded_total":704,"degraded_ratio":0.13068181818181818},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2023-07-07T06:01:05.781050+0000","services":{}},"progress_events":{}}

2023-07-07T17:34:03.714 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:02:05.737+0000 7fdc4ab1e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-07-07T06:02:05.928 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~88s
, 2023-07-07T06:02:06.025 INFO:teuthology.orchestra.run.smithi027.stdout:

2023-07-07T17:34:03.715 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:02:14.795+0000 7f4aa3027700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-07-07T06:02:14.800+0000 7f4aa3027700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-07-07T06:02:15.091 INFO:teuthology.orchestra.run.smithi027.stdout:

2023-07-07T17:34:03.716 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:02:23.724 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json
, 2023-07-07T06:02:23.850+0000 7fcfaaae6700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-07-07T06:02:23.856+0000 7fcfaaae6700 -1 WARNING: the following dangerous and experimental features are enabled: crimson

2023-07-07T17:34:03.718 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:02:32.767 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json
, 2023-07-07T06:02:32.893+0000 7f7590d47700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-07-07T06:02:32.899+0000 7f7590d47700 -1 WARNING: the following dangerous and experimental features are enabled: crimson

2023-07-07T17:34:03.719 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:02:41.818 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.462985+0000', 'last_change': '2023-07-07T06:00:59.462558+0000', 'last_active': '2023-07-07T06:00:59.462985+0000', 'last_peered': '2023-07-07T06:00:59.462985+0000', 'last_clean': '2023-07-07T06:00:25.415191+0000', 'last_became_active': '2023-07-07T06:00:59.462558+0000', 'last_became_peered': '2023-07-07T06:00:59.462558+0000', 'last_unstale': '2023-07-07T06:00:59.462985+0000', 'last_undegraded': '2023-07-07T06:00:59.462985+0000', 'last_fullsized': '2023-07-07T06:00:59.453964+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:02:41.819 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json
, 2023-07-07T06:02:41.944+0000 7fd8c412b700 -1 WARNING: the following dangerous and experimental features are enabled: crimson

2023-07-07T17:34:03.720 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:02:50.884 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean
, 2023-07-07T06:02:50.884 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.462985+0000', 'last_change': '2023-07-07T06:00:59.462558+0000', 'last_active': '2023-07-07T06:00:59.462985+0000', 'last_peered': '2023-07-07T06:00:59.462985+0000', 'last_clean': '2023-07-07T06:00:25.415191+0000', 'last_became_active': '2023-07-07T06:00:59.462558+0000', 'last_became_peered': '2023-07-07T06:00:59.462558+0000', 'last_unstale': '2023-07-07T06:00:59.462985+0000', 'last_undegraded': '2023-07-07T06:00:59.462985+0000', 'last_fullsized': '2023-07-07T06:00:59.453964+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:02:50.885 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json

2023-07-07T17:34:03.722 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:02:59.945 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean
, 2023-07-07T06:02:59.946 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.462985+0000', 'last_change': '2023-07-07T06:00:59.462558+0000', 'last_active': '2023-07-07T06:00:59.462985+0000', 'last_peered': '2023-07-07T06:00:59.462985+0000', 'last_clean': '2023-07-07T06:00:25.415191+0000', 'last_became_active': '2023-07-07T06:00:59.462558+0000', 'last_became_peered': '2023-07-07T06:00:59.462558+0000', 'last_unstale': '2023-07-07T06:00:59.462985+0000', 'last_undegraded': '2023-07-07T06:00:59.462985+0000', 'last_fullsized': '2023-07-07T06:00:59.453964+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:02:59.946 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json

2023-07-07T17:34:03.723 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:03:09.001 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.463127+0000', 'last_change': '2023-07-07T06:00:59.462748+0000', 'last_active': '2023-07-07T06:00:59.463127+0000', 'last_peered': '2023-07-07T06:00:59.463127+0000', 'last_clean': '2023-07-07T06:00:22.399960+0000', 'last_became_active': '2023-07-07T06:00:59.462748+0000', 'last_became_peered': '2023-07-07T06:00:59.462748+0000', 'last_unstale': '2023-07-07T06:00:59.463127+0000', 'last_undegraded': '2023-07-07T06:00:59.463127+0000', 'last_fullsized': '2023-07-07T06:00:59.455926+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []}
, 2023-07-07T06:03:09.002 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean
, 2023-07-07T06:03:09.002 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.462985+0000', 'last_change': '2023-07-07T06:00:59.462558+0000', 'last_active': '2023-07-07T06:00:59.462985+0000', 'last_peered': '2023-07-07T06:00:59.462985+0000', 'last_clean': '2023-07-07T06:00:25.415191+0000', 'last_became_active': '2023-07-07T06:00:59.462558+0000', 'last_became_peered': '2023-07-07T06:00:59.462558+0000', 'last_unstale': '2023-07-07T06:00:59.462985+0000', 'last_undegraded': '2023-07-07T06:00:59.462985+0000', 'last_fullsized': '2023-07-07T06:00:59.453964+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}

2023-07-07T17:34:03.724 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:03:18.048 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean
, 2023-07-07T06:03:18.049 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.463127+0000', 'last_change': '2023-07-07T06:00:59.462748+0000', 'last_active': '2023-07-07T06:00:59.463127+0000', 'last_peered': '2023-07-07T06:00:59.463127+0000', 'last_clean': '2023-07-07T06:00:22.399960+0000', 'last_became_active': '2023-07-07T06:00:59.462748+0000', 'last_became_peered': '2023-07-07T06:00:59.462748+0000', 'last_unstale': '2023-07-07T06:00:59.463127+0000', 'last_undegraded': '2023-07-07T06:00:59.463127+0000', 'last_fullsized': '2023-07-07T06:00:59.455926+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []}
, 2023-07-07T06:03:18.049 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean

2023-07-07T17:34:03.726 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:03:27.092 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean
, 2023-07-07T06:03:27.093 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.463127+0000', 'last_change': '2023-07-07T06:00:59.462748+0000', 'last_active': '2023-07-07T06:00:59.463127+0000', 'last_peered': '2023-07-07T06:00:59.463127+0000', 'last_clean': '2023-07-07T06:00:22.399960+0000', 'last_became_active': '2023-07-07T06:00:59.462748+0000', 'last_became_peered': '2023-07-07T06:00:59.462748+0000', 'last_unstale': '2023-07-07T06:00:59.463127+0000', 'last_undegraded': '2023-07-07T06:00:59.463127+0000', 'last_fullsized': '2023-07-07T06:00:59.455926+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []}
, 2023-07-07T06:03:27.093 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean

2023-07-07T17:34:03.727 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:03:36.152 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.463407+0000', 'last_change': '2023-07-07T06:00:59.462897+0000', 'last_active': '2023-07-07T06:00:59.463407+0000', 'last_peered': '2023-07-07T06:00:59.463407+0000', 'last_clean': '2023-07-07T06:00:08.305657+0000', 'last_became_active': '2023-07-07T06:00:59.462897+0000', 'last_became_peered': '2023-07-07T06:00:59.462897+0000', 'last_unstale': '2023-07-07T06:00:59.463407+0000', 'last_undegraded': '2023-07-07T06:00:59.463407+0000', 'last_fullsized': '2023-07-07T06:00:59.454340+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:03:36.152 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean
, 2023-07-07T06:03:36.152 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.463127+0000', 'last_change': '2023-07-07T06:00:59.462748+0000', 'last_active': '2023-07-07T06:00:59.463127+0000', 'last_peered': '2023-07-07T06:00:59.463127+0000', 'last_clean': '2023-07-07T06:00:22.399960+0000', 'last_became_active': '2023-07-07T06:00:59.462748+0000', 'last_became_peered': '2023-07-07T06:00:59.462748+0000', 'last_unstale': '2023-07-07T06:00:59.463127+0000', 'last_undegraded': '2023-07-07T06:00:59.463127+0000', 'last_fullsized': '2023-07-07T06:00:59.455926+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []}

2023-07-07T17:34:03.729 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:03:45.219 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean
, 2023-07-07T06:03:45.219 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.463407+0000', 'last_change': '2023-07-07T06:00:59.462897+0000', 'last_active': '2023-07-07T06:00:59.463407+0000', 'last_peered': '2023-07-07T06:00:59.463407+0000', 'last_clean': '2023-07-07T06:00:08.305657+0000', 'last_became_active': '2023-07-07T06:00:59.462897+0000', 'last_became_peered': '2023-07-07T06:00:59.462897+0000', 'last_unstale': '2023-07-07T06:00:59.463407+0000', 'last_undegraded': '2023-07-07T06:00:59.463407+0000', 'last_fullsized': '2023-07-07T06:00:59.454340+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:03:45.220 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean

2023-07-07T17:34:03.730 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:03:54.275 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean
, 2023-07-07T06:03:54.275 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.463407+0000', 'last_change': '2023-07-07T06:00:59.462897+0000', 'last_active': '2023-07-07T06:00:59.463407+0000', 'last_peered': '2023-07-07T06:00:59.463407+0000', 'last_clean': '2023-07-07T06:00:08.305657+0000', 'last_became_active': '2023-07-07T06:00:59.462897+0000', 'last_became_peered': '2023-07-07T06:00:59.462897+0000', 'last_unstale': '2023-07-07T06:00:59.463407+0000', 'last_undegraded': '2023-07-07T06:00:59.463407+0000', 'last_fullsized': '2023-07-07T06:00:59.454340+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:03:54.275 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean

2023-07-07T17:34:03.731 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:04:03.349 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2', 'version': "19'123", 'reported_seq': 192, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.480068+0000', 'last_change': '2023-07-07T06:00:59.470873+0000', 'last_active': '2023-07-07T06:00:59.480068+0000', 'last_peered': '2023-07-07T06:00:59.480068+0000', 'last_clean': '2023-07-07T06:00:35.066600+0000', 'last_became_active': '2023-07-07T06:00:59.470873+0000', 'last_became_peered': '2023-07-07T06:00:59.470873+0000', 'last_unstale': '2023-07-07T06:00:59.480068+0000', 'last_undegraded': '2023-07-07T06:00:59.462245+0000', 'last_fullsized': '2023-07-07T06:00:59.461717+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'objects_scrubbed': 0, 'log_size': 123, 'log_dups_size': 0, 'ondisk_log_size': 123, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 96893477, 'num_objects': 34, 'num_object_clones': 0, 'num_object_copies': 68, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 34, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 107, 'num_read_kb': 31202, 'num_write': 158, 'num_write_kb': 47074, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 34}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:04:03.349 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean
, 2023-07-07T06:04:03.349 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.463407+0000', 'last_change': '2023-07-07T06:00:59.462897+0000', 'last_active': '2023-07-07T06:00:59.463407+0000', 'last_peered': '2023-07-07T06:00:59.463407+0000', 'last_clean': '2023-07-07T06:00:08.305657+0000', 'last_became_active': '2023-07-07T06:00:59.462897+0000', 'last_became_peered': '2023-07-07T06:00:59.462897+0000', 'last_unstale': '2023-07-07T06:00:59.463407+0000', 'last_undegraded': '2023-07-07T06:00:59.463407+0000', 'last_fullsized': '2023-07-07T06:00:59.454340+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}

2023-07-07T17:34:03.733 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:04:12.418 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2', 'version': "19'123", 'reported_seq': 192, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.480068+0000', 'last_change': '2023-07-07T06:00:59.470873+0000', 'last_active': '2023-07-07T06:00:59.480068+0000', 'last_peered': '2023-07-07T06:00:59.480068+0000', 'last_clean': '2023-07-07T06:00:35.066600+0000', 'last_became_active': '2023-07-07T06:00:59.470873+0000', 'last_became_peered': '2023-07-07T06:00:59.470873+0000', 'last_unstale': '2023-07-07T06:00:59.480068+0000', 'last_undegraded': '2023-07-07T06:00:59.462245+0000', 'last_fullsized': '2023-07-07T06:00:59.461717+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'objects_scrubbed': 0, 'log_size': 123, 'log_dups_size': 0, 'ondisk_log_size': 123, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 96893477, 'num_objects': 34, 'num_object_clones': 0, 'num_object_copies': 68, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 34, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 107, 'num_read_kb': 31202, 'num_write': 158, 'num_write_kb': 47074, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 34}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:04:12.419 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean
, 2023-07-07T06:04:12.419 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.463407+0000', 'last_change': '2023-07-07T06:00:59.462897+0000', 'last_active': '2023-07-07T06:00:59.463407+0000', 'last_peered': '2023-07-07T06:00:59.463407+0000', 'last_clean': '2023-07-07T06:00:08.305657+0000', 'last_became_active': '2023-07-07T06:00:59.462897+0000', 'last_became_peered': '2023-07-07T06:00:59.462897+0000', 'last_unstale': '2023-07-07T06:00:59.463407+0000', 'last_undegraded': '2023-07-07T06:00:59.463407+0000', 'last_fullsized': '2023-07-07T06:00:59.454340+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}

2023-07-07T17:34:03.734 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:04:21.470 INFO:tasks.ceph.ceph_manager.ceph:PG 3.2 is not active+clean
, 2023-07-07T06:04:21.471 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2', 'version': "19'123", 'reported_seq': 192, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.480068+0000', 'last_change': '2023-07-07T06:00:59.470873+0000', 'last_active': '2023-07-07T06:00:59.480068+0000', 'last_peered': '2023-07-07T06:00:59.480068+0000', 'last_clean': '2023-07-07T06:00:35.066600+0000', 'last_became_active': '2023-07-07T06:00:59.470873+0000', 'last_became_peered': '2023-07-07T06:00:59.470873+0000', 'last_unstale': '2023-07-07T06:00:59.480068+0000', 'last_undegraded': '2023-07-07T06:00:59.462245+0000', 'last_fullsized': '2023-07-07T06:00:59.461717+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'objects_scrubbed': 0, 'log_size': 123, 'log_dups_size': 0, 'ondisk_log_size': 123, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 96893477, 'num_objects': 34, 'num_object_clones': 0, 'num_object_copies': 68, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 34, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 107, 'num_read_kb': 31202, 'num_write': 158, 'num_write_kb': 47074, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 34}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:04:21.471 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean

2023-07-07T17:34:03.735 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:04:30.516 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "14'97", 'reported_seq': 155, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.469971+0000', 'last_change': '2023-07-07T06:00:59.469575+0000', 'last_active': '2023-07-07T06:00:59.469971+0000', 'last_peered': '2023-07-07T06:00:59.469971+0000', 'last_clean': '2023-07-07T06:00:23.369029+0000', 'last_became_active': '2023-07-07T06:00:59.469575+0000', 'last_became_peered': '2023-07-07T06:00:59.469575+0000', 'last_unstale': '2023-07-07T06:00:59.469971+0000', 'last_undegraded': '2023-07-07T06:00:59.465049+0000', 'last_fullsized': '2023-07-07T06:00:59.464546+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []}
, 2023-07-07T06:04:30.516 INFO:tasks.ceph.ceph_manager.ceph:PG 3.2 is not active+clean
, 2023-07-07T06:04:30.517 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2', 'version': "19'123", 'reported_seq': 192, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.480068+0000', 'last_change': '2023-07-07T06:00:59.470873+0000', 'last_active': '2023-07-07T06:00:59.480068+0000', 'last_peered': '2023-07-07T06:00:59.480068+0000', 'last_clean': '2023-07-07T06:00:35.066600+0000', 'last_became_active': '2023-07-07T06:00:59.470873+0000', 'last_became_peered': '2023-07-07T06:00:59.470873+0000', 'last_unstale': '2023-07-07T06:00:59.480068+0000', 'last_undegraded': '2023-07-07T06:00:59.462245+0000', 'last_fullsized': '2023-07-07T06:00:59.461717+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'objects_scrubbed': 0, 'log_size': 123, 'log_dups_size': 0, 'ondisk_log_size': 123, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 96893477, 'num_objects': 34, 'num_object_clones': 0, 'num_object_copies': 68, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 34, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 107, 'num_read_kb': 31202, 'num_write': 158, 'num_write_kb': 47074, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 34}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}

2023-07-07T17:34:03.737 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:04:39.576 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "14'97", 'reported_seq': 155, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.469971+0000', 'last_change': '2023-07-07T06:00:59.469575+0000', 'last_active': '2023-07-07T06:00:59.469971+0000', 'last_peered': '2023-07-07T06:00:59.469971+0000', 'last_clean': '2023-07-07T06:00:23.369029+0000', 'last_became_active': '2023-07-07T06:00:59.469575+0000', 'last_became_peered': '2023-07-07T06:00:59.469575+0000', 'last_unstale': '2023-07-07T06:00:59.469971+0000', 'last_undegraded': '2023-07-07T06:00:59.465049+0000', 'last_fullsized': '2023-07-07T06:00:59.464546+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []}
, 2023-07-07T06:04:39.576 INFO:tasks.ceph.ceph_manager.ceph:PG 3.2 is not active+clean
, 2023-07-07T06:04:39.577 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2', 'version': "19'123", 'reported_seq': 192, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.480068+0000', 'last_change': '2023-07-07T06:00:59.470873+0000', 'last_active': '2023-07-07T06:00:59.480068+0000', 'last_peered': '2023-07-07T06:00:59.480068+0000', 'last_clean': '2023-07-07T06:00:35.066600+0000', 'last_became_active': '2023-07-07T06:00:59.470873+0000', 'last_became_peered': '2023-07-07T06:00:59.470873+0000', 'last_unstale': '2023-07-07T06:00:59.480068+0000', 'last_undegraded': '2023-07-07T06:00:59.462245+0000', 'last_fullsized': '2023-07-07T06:00:59.461717+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'objects_scrubbed': 0, 'log_size': 123, 'log_dups_size': 0, 'ondisk_log_size': 123, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 96893477, 'num_objects': 34, 'num_object_clones': 0, 'num_object_copies': 68, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 34, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 107, 'num_read_kb': 31202, 'num_write': 158, 'num_write_kb': 47074, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 34}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}

2023-07-07T17:34:03.738 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:04:48.624 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean
, 2023-07-07T06:04:48.624 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "14'97", 'reported_seq': 155, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.469971+0000', 'last_change': '2023-07-07T06:00:59.469575+0000', 'last_active': '2023-07-07T06:00:59.469971+0000', 'last_peered': '2023-07-07T06:00:59.469971+0000', 'last_clean': '2023-07-07T06:00:23.369029+0000', 'last_became_active': '2023-07-07T06:00:59.469575+0000', 'last_became_peered': '2023-07-07T06:00:59.469575+0000', 'last_unstale': '2023-07-07T06:00:59.469971+0000', 'last_undegraded': '2023-07-07T06:00:59.465049+0000', 'last_fullsized': '2023-07-07T06:00:59.464546+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []}
, 2023-07-07T06:04:48.625 INFO:tasks.ceph.ceph_manager.ceph:PG 3.2 is not active+clean

2023-07-07T17:34:03.740 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:04:57.681 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.468569+0000', 'last_change': '2023-07-07T06:00:59.468122+0000', 'last_active': '2023-07-07T06:00:59.468569+0000', 'last_peered': '2023-07-07T06:00:59.468569+0000', 'last_clean': '2023-07-07T06:00:25.415275+0000', 'last_became_active': '2023-07-07T06:00:59.468122+0000', 'last_became_peered': '2023-07-07T06:00:59.468122+0000', 'last_unstale': '2023-07-07T06:00:59.468569+0000', 'last_undegraded': '2023-07-07T06:00:59.468569+0000', 'last_fullsized': '2023-07-07T06:00:59.457597+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:04:57.682 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean
, 2023-07-07T06:04:57.682 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "14'97", 'reported_seq': 155, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.469971+0000', 'last_change': '2023-07-07T06:00:59.469575+0000', 'last_active': '2023-07-07T06:00:59.469971+0000', 'last_peered': '2023-07-07T06:00:59.469971+0000', 'last_clean': '2023-07-07T06:00:23.369029+0000', 'last_became_active': '2023-07-07T06:00:59.469575+0000', 'last_became_peered': '2023-07-07T06:00:59.469575+0000', 'last_unstale': '2023-07-07T06:00:59.469971+0000', 'last_undegraded': '2023-07-07T06:00:59.465049+0000', 'last_fullsized': '2023-07-07T06:00:59.464546+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []}

2023-07-07T17:34:03.741 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:05:06.737 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.468569+0000', 'last_change': '2023-07-07T06:00:59.468122+0000', 'last_active': '2023-07-07T06:00:59.468569+0000', 'last_peered': '2023-07-07T06:00:59.468569+0000', 'last_clean': '2023-07-07T06:00:25.415275+0000', 'last_became_active': '2023-07-07T06:00:59.468122+0000', 'last_became_peered': '2023-07-07T06:00:59.468122+0000', 'last_unstale': '2023-07-07T06:00:59.468569+0000', 'last_undegraded': '2023-07-07T06:00:59.468569+0000', 'last_fullsized': '2023-07-07T06:00:59.457597+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:05:06.738 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean
, 2023-07-07T06:05:06.738 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "14'97", 'reported_seq': 155, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.469971+0000', 'last_change': '2023-07-07T06:00:59.469575+0000', 'last_active': '2023-07-07T06:00:59.469971+0000', 'last_peered': '2023-07-07T06:00:59.469971+0000', 'last_clean': '2023-07-07T06:00:23.369029+0000', 'last_became_active': '2023-07-07T06:00:59.469575+0000', 'last_became_peered': '2023-07-07T06:00:59.469575+0000', 'last_unstale': '2023-07-07T06:00:59.469971+0000', 'last_undegraded': '2023-07-07T06:00:59.465049+0000', 'last_fullsized': '2023-07-07T06:00:59.464546+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []}

2023-07-07T17:34:03.742 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:05:15.804 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean
, 2023-07-07T06:05:15.805 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.468569+0000', 'last_change': '2023-07-07T06:00:59.468122+0000', 'last_active': '2023-07-07T06:00:59.468569+0000', 'last_peered': '2023-07-07T06:00:59.468569+0000', 'last_clean': '2023-07-07T06:00:25.415275+0000', 'last_became_active': '2023-07-07T06:00:59.468122+0000', 'last_became_peered': '2023-07-07T06:00:59.468122+0000', 'last_unstale': '2023-07-07T06:00:59.468569+0000', 'last_undegraded': '2023-07-07T06:00:59.468569+0000', 'last_fullsized': '2023-07-07T06:00:59.457597+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:05:15.805 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean

2023-07-07T17:34:03.744 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:05:24.890 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.3', 'version': "21'107", 'reported_seq': 156, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.478255+0000', 'last_change': '2023-07-07T06:00:59.464604+0000', 'last_active': '2023-07-07T06:00:59.478255+0000', 'last_peered': '2023-07-07T06:00:59.478255+0000', 'last_clean': '2023-07-07T06:00:33.546473+0000', 'last_became_active': '2023-07-07T06:00:59.464604+0000', 'last_became_peered': '2023-07-07T06:00:59.464604+0000', 'last_unstale': '2023-07-07T06:00:59.478255+0000', 'last_undegraded': '2023-07-07T06:00:59.458856+0000', 'last_fullsized': '2023-07-07T06:00:59.458231+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'objects_scrubbed': 0, 'log_size': 107, 'log_dups_size': 0, 'ondisk_log_size': 107, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 82623806, 'num_objects': 29, 'num_object_clones': 0, 'num_object_copies': 58, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 29, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 61, 'num_read_kb': 31, 'num_write': 138, 'num_write_kb': 39292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 29}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []}
, 2023-07-07T06:05:24.891 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean
, 2023-07-07T06:05:24.891 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.468569+0000', 'last_change': '2023-07-07T06:00:59.468122+0000', 'last_active': '2023-07-07T06:00:59.468569+0000', 'last_peered': '2023-07-07T06:00:59.468569+0000', 'last_clean': '2023-07-07T06:00:25.415275+0000', 'last_became_active': '2023-07-07T06:00:59.468122+0000', 'last_became_peered': '2023-07-07T06:00:59.468122+0000', 'last_unstale': '2023-07-07T06:00:59.468569+0000', 'last_undegraded': '2023-07-07T06:00:59.468569+0000', 'last_fullsized': '2023-07-07T06:00:59.457597+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}

2023-07-07T17:34:03.745 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:05:33.966 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.3', 'version': "21'107", 'reported_seq': 156, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.478255+0000', 'last_change': '2023-07-07T06:00:59.464604+0000', 'last_active': '2023-07-07T06:00:59.478255+0000', 'last_peered': '2023-07-07T06:00:59.478255+0000', 'last_clean': '2023-07-07T06:00:33.546473+0000', 'last_became_active': '2023-07-07T06:00:59.464604+0000', 'last_became_peered': '2023-07-07T06:00:59.464604+0000', 'last_unstale': '2023-07-07T06:00:59.478255+0000', 'last_undegraded': '2023-07-07T06:00:59.458856+0000', 'last_fullsized': '2023-07-07T06:00:59.458231+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'objects_scrubbed': 0, 'log_size': 107, 'log_dups_size': 0, 'ondisk_log_size': 107, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 82623806, 'num_objects': 29, 'num_object_clones': 0, 'num_object_copies': 58, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 29, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 61, 'num_read_kb': 31, 'num_write': 138, 'num_write_kb': 39292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 29}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []}
, 2023-07-07T06:05:33.967 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean
, 2023-07-07T06:05:33.967 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.468569+0000', 'last_change': '2023-07-07T06:00:59.468122+0000', 'last_active': '2023-07-07T06:00:59.468569+0000', 'last_peered': '2023-07-07T06:00:59.468569+0000', 'last_clean': '2023-07-07T06:00:25.415275+0000', 'last_became_active': '2023-07-07T06:00:59.468122+0000', 'last_became_peered': '2023-07-07T06:00:59.468122+0000', 'last_unstale': '2023-07-07T06:00:59.468569+0000', 'last_undegraded': '2023-07-07T06:00:59.468569+0000', 'last_fullsized': '2023-07-07T06:00:59.457597+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}

2023-07-07T17:34:03.745 WARNING:scrape:Ignoring malparsed backtrace:   File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception
,   File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise
,     raise value.with_traceback(tb)

2023-07-07T17:34:03.746 WARNING:scrape:Ignoring malparsed backtrace: Traceback (most recent call last):
,   File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/daemon/state.py", line 139, in stop
,     run.wait([self.proc], timeout=timeout)

2023-07-07T17:34:03.746 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 479, in wait
,     proc.wait()
,   File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait

2023-07-07T17:34:03.747 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait
,     self._raise_for_status()
,   File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status

2023-07-07T17:34:03.747 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.115332.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116271.log
,  49.6% -- replaced with /var/log/ceph/ceph-client.admin.115332.log.gz
, /var/log/ceph/ceph-client.admin.121237.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114695.log

2023-07-07T17:34:03.748 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.123891.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120395.log
,  49.2% -- replaced with /var/log/ceph/ceph-client.admin.123891.log.gz
, /var/log/ceph/ceph-client.admin.124136.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121678.log

2023-07-07T17:34:03.748 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.123695.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120698.log
,  49.6% -- replaced with /var/log/ceph/ceph-client.admin.123695.log.gz
, /var/log/ceph/ceph-client.admin.117300.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113421.log

2023-07-07T17:34:03.748 WARNING:scrape:Ignoring malparsed backtrace:  50.0% -- replaced with /var/log/ceph/ceph-client.admin.114303.log.gz
, gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122217.log
, /var/log/ceph/ceph-client.admin.123001.log:	 49.6% -- replaced with /var/log/ceph/ceph-client.admin.123001.log.gz/var/log/ceph/ceph-client.admin.111201.log:

2023-07-07T17:34:03.749 WARNING:scrape:Ignoring malparsed backtrace:  49.6% -- replaced with /var/log/ceph/ceph-client.admin.117643.log.gz
, /var/log/ceph/ceph-client.admin.122119.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123940.log
,  49.6% -- replaced with /var/log/ceph/ceph-client.admin.122119.log.gz

2023-07-07T17:34:03.749 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.121531.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118288.log
,  50.0% -- replaced with /var/log/ceph/ceph-client.admin.121531.log.gz
, /var/log/ceph/ceph-client.admin.118974.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115781.log

2023-07-07T17:34:03.750 WARNING:scrape:Ignoring malparsed backtrace: Traceback (most recent call last):
,   File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run_tasks.py", line 192, in run_tasks
,     suppress = manager.__exit__(*exc_info)

2023-07-07T17:34:03.750 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:07:45.041 DEBUG:teuthology.orchestra.run.smithi027:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz
, 2023-07-07T06:07:45.063 DEBUG:teuthology.orchestra.run.smithi167:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz
, 2023-07-07T06:07:45.145 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo

2023-07-07T17:34:03.772 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.0.0-4717-g6e5383fd (6e5383fdd6d402108195ddbbefb66fca2cebcf96) reef (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7efc469a6ed4]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x61b) [0x55e7f1e48535]

2023-07-07T17:34:03.775 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:01:02.676 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json
, 2023-07-07T06:01:02.804+0000 7f22ab646700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-07-07T06:01:02.810+0000 7f22ab646700 -1 WARNING: the following dangerous and experimental features are enabled: crimson

2023-07-07T17:34:03.776 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:01:11.727 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now
, 2023-07-07T06:01:11.727 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json
, 2023-07-07T06:01:11.853+0000 7f8496fe9700 -1 WARNING: the following dangerous and experimental features are enabled: crimson

2023-07-07T17:34:03.777 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:01:20.787 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now
, 2023-07-07T06:01:20.788 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json
, 2023-07-07T06:01:20.914+0000 7fe12ceba700 -1 WARNING: the following dangerous and experimental features are enabled: crimson

2023-07-07T17:34:03.779 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:01:29.826 INFO:teuthology.orchestra.run.smithi027.stdout:{"fsid":"421db855-4d57-41f4-bef0-9c6557345e90","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 92/704 objects degraded (13.068%), 4 pgs degraded","count":4},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":92,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":21,"num_osds":4,"num_up_osds":3,"osd_up_since":1688709658,"num_in_osds":4,"osd_in_since":1688709599,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+undersized","count":9},{"state_name":"active+undersized+degraded","count":4}],"num_pgs":25,"num_pools":3,"num_objects":352,"data_bytes":1022632651,"bytes_used":1450606592,"bytes_avail":428046123008,"bytes_total":429496729600,"degraded_objects":92,"degraded_total":704,"degraded_ratio":0.13068181818181818},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2023-07-07T06:01:05.781050+0000","services":{}},"progress_events":{}}
, 2023-07-07T06:01:29.840 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now
, 2023-07-07T06:01:29.841 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json

2023-07-07T17:34:03.780 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:01:38.874 INFO:teuthology.orchestra.run.smithi027.stdout:
, 2023-07-07T06:01:38.874 INFO:teuthology.orchestra.run.smithi027.stdout:{"fsid":"421db855-4d57-41f4-bef0-9c6557345e90","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 92/704 objects degraded (13.068%), 4 pgs degraded","count":4},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":101,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":21,"num_osds":4,"num_up_osds":3,"osd_up_since":1688709658,"num_in_osds":4,"osd_in_since":1688709599,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+undersized","count":9},{"state_name":"active+undersized+degraded","count":4}],"num_pgs":25,"num_pools":3,"num_objects":352,"data_bytes":1022632651,"bytes_used":1450606592,"bytes_avail":428046123008,"bytes_total":429496729600,"degraded_objects":92,"degraded_total":704,"degraded_ratio":0.13068181818181818},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2023-07-07T06:01:05.781050+0000","services":{}},"progress_events":{}}
, 2023-07-07T06:01:38.888 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now

2023-07-07T17:34:03.782 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:01:47.925 INFO:teuthology.orchestra.run.smithi027.stdout:
, 2023-07-07T06:01:47.925 INFO:teuthology.orchestra.run.smithi027.stdout:{"fsid":"421db855-4d57-41f4-bef0-9c6557345e90","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 92/704 objects degraded (13.068%), 4 pgs degraded","count":4},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":110,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":21,"num_osds":4,"num_up_osds":3,"osd_up_since":1688709658,"num_in_osds":4,"osd_in_since":1688709599,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+undersized","count":9},{"state_name":"active+undersized+degraded","count":4}],"num_pgs":25,"num_pools":3,"num_objects":352,"data_bytes":1022632651,"bytes_used":1450606592,"bytes_avail":428046123008,"bytes_total":429496729600,"degraded_objects":92,"degraded_total":704,"degraded_ratio":0.13068181818181818},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2023-07-07T06:01:05.781050+0000","services":{}},"progress_events":{}}
, 2023-07-07T06:01:47.940 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now

2023-07-07T17:34:03.783 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:01:56.693+0000 7f654f336700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-07-07T06:01:56.982 INFO:teuthology.orchestra.run.smithi027.stdout:
, 2023-07-07T06:01:56.982 INFO:teuthology.orchestra.run.smithi027.stdout:{"fsid":"421db855-4d57-41f4-bef0-9c6557345e90","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 92/704 objects degraded (13.068%), 4 pgs degraded","count":4},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":119,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":21,"num_osds":4,"num_up_osds":3,"osd_up_since":1688709658,"num_in_osds":4,"osd_in_since":1688709599,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+undersized","count":9},{"state_name":"active+undersized+degraded","count":4}],"num_pgs":25,"num_pools":3,"num_objects":352,"data_bytes":1022632651,"bytes_used":1450606592,"bytes_avail":428046123008,"bytes_total":429496729600,"degraded_objects":92,"degraded_total":704,"degraded_ratio":0.13068181818181818},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2023-07-07T06:01:05.781050+0000","services":{}},"progress_events":{}}

2023-07-07T17:34:03.784 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:02:05.737+0000 7fdc4ab1e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-07-07T06:02:05.928 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~88s
, 2023-07-07T06:02:06.025 INFO:teuthology.orchestra.run.smithi027.stdout:

2023-07-07T17:34:03.786 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:02:14.795+0000 7f4aa3027700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-07-07T06:02:14.800+0000 7f4aa3027700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-07-07T06:02:15.091 INFO:teuthology.orchestra.run.smithi027.stdout:

2023-07-07T17:34:03.787 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:02:23.724 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json
, 2023-07-07T06:02:23.850+0000 7fcfaaae6700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-07-07T06:02:23.856+0000 7fcfaaae6700 -1 WARNING: the following dangerous and experimental features are enabled: crimson

2023-07-07T17:34:03.789 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:02:32.767 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json
, 2023-07-07T06:02:32.893+0000 7f7590d47700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-07-07T06:02:32.899+0000 7f7590d47700 -1 WARNING: the following dangerous and experimental features are enabled: crimson

2023-07-07T17:34:03.790 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:02:41.818 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.462985+0000', 'last_change': '2023-07-07T06:00:59.462558+0000', 'last_active': '2023-07-07T06:00:59.462985+0000', 'last_peered': '2023-07-07T06:00:59.462985+0000', 'last_clean': '2023-07-07T06:00:25.415191+0000', 'last_became_active': '2023-07-07T06:00:59.462558+0000', 'last_became_peered': '2023-07-07T06:00:59.462558+0000', 'last_unstale': '2023-07-07T06:00:59.462985+0000', 'last_undegraded': '2023-07-07T06:00:59.462985+0000', 'last_fullsized': '2023-07-07T06:00:59.453964+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:02:41.819 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json
, 2023-07-07T06:02:41.944+0000 7fd8c412b700 -1 WARNING: the following dangerous and experimental features are enabled: crimson

2023-07-07T17:34:03.791 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:02:50.884 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean
, 2023-07-07T06:02:50.884 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.462985+0000', 'last_change': '2023-07-07T06:00:59.462558+0000', 'last_active': '2023-07-07T06:00:59.462985+0000', 'last_peered': '2023-07-07T06:00:59.462985+0000', 'last_clean': '2023-07-07T06:00:25.415191+0000', 'last_became_active': '2023-07-07T06:00:59.462558+0000', 'last_became_peered': '2023-07-07T06:00:59.462558+0000', 'last_unstale': '2023-07-07T06:00:59.462985+0000', 'last_undegraded': '2023-07-07T06:00:59.462985+0000', 'last_fullsized': '2023-07-07T06:00:59.453964+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:02:50.885 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json

2023-07-07T17:34:03.793 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:02:59.945 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean
, 2023-07-07T06:02:59.946 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.462985+0000', 'last_change': '2023-07-07T06:00:59.462558+0000', 'last_active': '2023-07-07T06:00:59.462985+0000', 'last_peered': '2023-07-07T06:00:59.462985+0000', 'last_clean': '2023-07-07T06:00:25.415191+0000', 'last_became_active': '2023-07-07T06:00:59.462558+0000', 'last_became_peered': '2023-07-07T06:00:59.462558+0000', 'last_unstale': '2023-07-07T06:00:59.462985+0000', 'last_undegraded': '2023-07-07T06:00:59.462985+0000', 'last_fullsized': '2023-07-07T06:00:59.453964+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:02:59.946 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json

2023-07-07T17:34:03.794 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:03:09.001 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.463127+0000', 'last_change': '2023-07-07T06:00:59.462748+0000', 'last_active': '2023-07-07T06:00:59.463127+0000', 'last_peered': '2023-07-07T06:00:59.463127+0000', 'last_clean': '2023-07-07T06:00:22.399960+0000', 'last_became_active': '2023-07-07T06:00:59.462748+0000', 'last_became_peered': '2023-07-07T06:00:59.462748+0000', 'last_unstale': '2023-07-07T06:00:59.463127+0000', 'last_undegraded': '2023-07-07T06:00:59.463127+0000', 'last_fullsized': '2023-07-07T06:00:59.455926+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []}
, 2023-07-07T06:03:09.002 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean
, 2023-07-07T06:03:09.002 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.462985+0000', 'last_change': '2023-07-07T06:00:59.462558+0000', 'last_active': '2023-07-07T06:00:59.462985+0000', 'last_peered': '2023-07-07T06:00:59.462985+0000', 'last_clean': '2023-07-07T06:00:25.415191+0000', 'last_became_active': '2023-07-07T06:00:59.462558+0000', 'last_became_peered': '2023-07-07T06:00:59.462558+0000', 'last_unstale': '2023-07-07T06:00:59.462985+0000', 'last_undegraded': '2023-07-07T06:00:59.462985+0000', 'last_fullsized': '2023-07-07T06:00:59.453964+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}

2023-07-07T17:34:03.796 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:03:18.048 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean
, 2023-07-07T06:03:18.049 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.463127+0000', 'last_change': '2023-07-07T06:00:59.462748+0000', 'last_active': '2023-07-07T06:00:59.463127+0000', 'last_peered': '2023-07-07T06:00:59.463127+0000', 'last_clean': '2023-07-07T06:00:22.399960+0000', 'last_became_active': '2023-07-07T06:00:59.462748+0000', 'last_became_peered': '2023-07-07T06:00:59.462748+0000', 'last_unstale': '2023-07-07T06:00:59.463127+0000', 'last_undegraded': '2023-07-07T06:00:59.463127+0000', 'last_fullsized': '2023-07-07T06:00:59.455926+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []}
, 2023-07-07T06:03:18.049 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean

2023-07-07T17:34:03.797 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:03:27.092 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean
, 2023-07-07T06:03:27.093 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.463127+0000', 'last_change': '2023-07-07T06:00:59.462748+0000', 'last_active': '2023-07-07T06:00:59.463127+0000', 'last_peered': '2023-07-07T06:00:59.463127+0000', 'last_clean': '2023-07-07T06:00:22.399960+0000', 'last_became_active': '2023-07-07T06:00:59.462748+0000', 'last_became_peered': '2023-07-07T06:00:59.462748+0000', 'last_unstale': '2023-07-07T06:00:59.463127+0000', 'last_undegraded': '2023-07-07T06:00:59.463127+0000', 'last_fullsized': '2023-07-07T06:00:59.455926+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []}
, 2023-07-07T06:03:27.093 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean

2023-07-07T17:34:03.799 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:03:36.152 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.463407+0000', 'last_change': '2023-07-07T06:00:59.462897+0000', 'last_active': '2023-07-07T06:00:59.463407+0000', 'last_peered': '2023-07-07T06:00:59.463407+0000', 'last_clean': '2023-07-07T06:00:08.305657+0000', 'last_became_active': '2023-07-07T06:00:59.462897+0000', 'last_became_peered': '2023-07-07T06:00:59.462897+0000', 'last_unstale': '2023-07-07T06:00:59.463407+0000', 'last_undegraded': '2023-07-07T06:00:59.463407+0000', 'last_fullsized': '2023-07-07T06:00:59.454340+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:03:36.152 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean
, 2023-07-07T06:03:36.152 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.463127+0000', 'last_change': '2023-07-07T06:00:59.462748+0000', 'last_active': '2023-07-07T06:00:59.463127+0000', 'last_peered': '2023-07-07T06:00:59.463127+0000', 'last_clean': '2023-07-07T06:00:22.399960+0000', 'last_became_active': '2023-07-07T06:00:59.462748+0000', 'last_became_peered': '2023-07-07T06:00:59.462748+0000', 'last_unstale': '2023-07-07T06:00:59.463127+0000', 'last_undegraded': '2023-07-07T06:00:59.463127+0000', 'last_fullsized': '2023-07-07T06:00:59.455926+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []}

2023-07-07T17:34:03.800 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:03:45.219 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean
, 2023-07-07T06:03:45.219 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.463407+0000', 'last_change': '2023-07-07T06:00:59.462897+0000', 'last_active': '2023-07-07T06:00:59.463407+0000', 'last_peered': '2023-07-07T06:00:59.463407+0000', 'last_clean': '2023-07-07T06:00:08.305657+0000', 'last_became_active': '2023-07-07T06:00:59.462897+0000', 'last_became_peered': '2023-07-07T06:00:59.462897+0000', 'last_unstale': '2023-07-07T06:00:59.463407+0000', 'last_undegraded': '2023-07-07T06:00:59.463407+0000', 'last_fullsized': '2023-07-07T06:00:59.454340+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:03:45.220 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean

2023-07-07T17:34:03.801 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:03:54.275 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean
, 2023-07-07T06:03:54.275 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.463407+0000', 'last_change': '2023-07-07T06:00:59.462897+0000', 'last_active': '2023-07-07T06:00:59.463407+0000', 'last_peered': '2023-07-07T06:00:59.463407+0000', 'last_clean': '2023-07-07T06:00:08.305657+0000', 'last_became_active': '2023-07-07T06:00:59.462897+0000', 'last_became_peered': '2023-07-07T06:00:59.462897+0000', 'last_unstale': '2023-07-07T06:00:59.463407+0000', 'last_undegraded': '2023-07-07T06:00:59.463407+0000', 'last_fullsized': '2023-07-07T06:00:59.454340+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:03:54.275 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean

2023-07-07T17:34:03.803 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:04:03.349 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2', 'version': "19'123", 'reported_seq': 192, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.480068+0000', 'last_change': '2023-07-07T06:00:59.470873+0000', 'last_active': '2023-07-07T06:00:59.480068+0000', 'last_peered': '2023-07-07T06:00:59.480068+0000', 'last_clean': '2023-07-07T06:00:35.066600+0000', 'last_became_active': '2023-07-07T06:00:59.470873+0000', 'last_became_peered': '2023-07-07T06:00:59.470873+0000', 'last_unstale': '2023-07-07T06:00:59.480068+0000', 'last_undegraded': '2023-07-07T06:00:59.462245+0000', 'last_fullsized': '2023-07-07T06:00:59.461717+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'objects_scrubbed': 0, 'log_size': 123, 'log_dups_size': 0, 'ondisk_log_size': 123, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 96893477, 'num_objects': 34, 'num_object_clones': 0, 'num_object_copies': 68, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 34, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 107, 'num_read_kb': 31202, 'num_write': 158, 'num_write_kb': 47074, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 34}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:04:03.349 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean
, 2023-07-07T06:04:03.349 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.463407+0000', 'last_change': '2023-07-07T06:00:59.462897+0000', 'last_active': '2023-07-07T06:00:59.463407+0000', 'last_peered': '2023-07-07T06:00:59.463407+0000', 'last_clean': '2023-07-07T06:00:08.305657+0000', 'last_became_active': '2023-07-07T06:00:59.462897+0000', 'last_became_peered': '2023-07-07T06:00:59.462897+0000', 'last_unstale': '2023-07-07T06:00:59.463407+0000', 'last_undegraded': '2023-07-07T06:00:59.463407+0000', 'last_fullsized': '2023-07-07T06:00:59.454340+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}

2023-07-07T17:34:03.804 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:04:12.418 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2', 'version': "19'123", 'reported_seq': 192, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.480068+0000', 'last_change': '2023-07-07T06:00:59.470873+0000', 'last_active': '2023-07-07T06:00:59.480068+0000', 'last_peered': '2023-07-07T06:00:59.480068+0000', 'last_clean': '2023-07-07T06:00:35.066600+0000', 'last_became_active': '2023-07-07T06:00:59.470873+0000', 'last_became_peered': '2023-07-07T06:00:59.470873+0000', 'last_unstale': '2023-07-07T06:00:59.480068+0000', 'last_undegraded': '2023-07-07T06:00:59.462245+0000', 'last_fullsized': '2023-07-07T06:00:59.461717+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'objects_scrubbed': 0, 'log_size': 123, 'log_dups_size': 0, 'ondisk_log_size': 123, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 96893477, 'num_objects': 34, 'num_object_clones': 0, 'num_object_copies': 68, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 34, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 107, 'num_read_kb': 31202, 'num_write': 158, 'num_write_kb': 47074, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 34}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:04:12.419 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean
, 2023-07-07T06:04:12.419 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 8, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.463407+0000', 'last_change': '2023-07-07T06:00:59.462897+0000', 'last_active': '2023-07-07T06:00:59.463407+0000', 'last_peered': '2023-07-07T06:00:59.463407+0000', 'last_clean': '2023-07-07T06:00:08.305657+0000', 'last_became_active': '2023-07-07T06:00:59.462897+0000', 'last_became_peered': '2023-07-07T06:00:59.462897+0000', 'last_unstale': '2023-07-07T06:00:59.463407+0000', 'last_undegraded': '2023-07-07T06:00:59.463407+0000', 'last_fullsized': '2023-07-07T06:00:59.454340+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}

2023-07-07T17:34:03.806 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:04:21.470 INFO:tasks.ceph.ceph_manager.ceph:PG 3.2 is not active+clean
, 2023-07-07T06:04:21.471 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2', 'version': "19'123", 'reported_seq': 192, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.480068+0000', 'last_change': '2023-07-07T06:00:59.470873+0000', 'last_active': '2023-07-07T06:00:59.480068+0000', 'last_peered': '2023-07-07T06:00:59.480068+0000', 'last_clean': '2023-07-07T06:00:35.066600+0000', 'last_became_active': '2023-07-07T06:00:59.470873+0000', 'last_became_peered': '2023-07-07T06:00:59.470873+0000', 'last_unstale': '2023-07-07T06:00:59.480068+0000', 'last_undegraded': '2023-07-07T06:00:59.462245+0000', 'last_fullsized': '2023-07-07T06:00:59.461717+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'objects_scrubbed': 0, 'log_size': 123, 'log_dups_size': 0, 'ondisk_log_size': 123, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 96893477, 'num_objects': 34, 'num_object_clones': 0, 'num_object_copies': 68, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 34, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 107, 'num_read_kb': 31202, 'num_write': 158, 'num_write_kb': 47074, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 34}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:04:21.471 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean

2023-07-07T17:34:03.807 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:04:30.516 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "14'97", 'reported_seq': 155, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.469971+0000', 'last_change': '2023-07-07T06:00:59.469575+0000', 'last_active': '2023-07-07T06:00:59.469971+0000', 'last_peered': '2023-07-07T06:00:59.469971+0000', 'last_clean': '2023-07-07T06:00:23.369029+0000', 'last_became_active': '2023-07-07T06:00:59.469575+0000', 'last_became_peered': '2023-07-07T06:00:59.469575+0000', 'last_unstale': '2023-07-07T06:00:59.469971+0000', 'last_undegraded': '2023-07-07T06:00:59.465049+0000', 'last_fullsized': '2023-07-07T06:00:59.464546+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []}
, 2023-07-07T06:04:30.516 INFO:tasks.ceph.ceph_manager.ceph:PG 3.2 is not active+clean
, 2023-07-07T06:04:30.517 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2', 'version': "19'123", 'reported_seq': 192, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.480068+0000', 'last_change': '2023-07-07T06:00:59.470873+0000', 'last_active': '2023-07-07T06:00:59.480068+0000', 'last_peered': '2023-07-07T06:00:59.480068+0000', 'last_clean': '2023-07-07T06:00:35.066600+0000', 'last_became_active': '2023-07-07T06:00:59.470873+0000', 'last_became_peered': '2023-07-07T06:00:59.470873+0000', 'last_unstale': '2023-07-07T06:00:59.480068+0000', 'last_undegraded': '2023-07-07T06:00:59.462245+0000', 'last_fullsized': '2023-07-07T06:00:59.461717+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'objects_scrubbed': 0, 'log_size': 123, 'log_dups_size': 0, 'ondisk_log_size': 123, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 96893477, 'num_objects': 34, 'num_object_clones': 0, 'num_object_copies': 68, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 34, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 107, 'num_read_kb': 31202, 'num_write': 158, 'num_write_kb': 47074, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 34}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}

2023-07-07T17:34:03.809 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:04:39.576 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "14'97", 'reported_seq': 155, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.469971+0000', 'last_change': '2023-07-07T06:00:59.469575+0000', 'last_active': '2023-07-07T06:00:59.469971+0000', 'last_peered': '2023-07-07T06:00:59.469971+0000', 'last_clean': '2023-07-07T06:00:23.369029+0000', 'last_became_active': '2023-07-07T06:00:59.469575+0000', 'last_became_peered': '2023-07-07T06:00:59.469575+0000', 'last_unstale': '2023-07-07T06:00:59.469971+0000', 'last_undegraded': '2023-07-07T06:00:59.465049+0000', 'last_fullsized': '2023-07-07T06:00:59.464546+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []}
, 2023-07-07T06:04:39.576 INFO:tasks.ceph.ceph_manager.ceph:PG 3.2 is not active+clean
, 2023-07-07T06:04:39.577 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2', 'version': "19'123", 'reported_seq': 192, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.480068+0000', 'last_change': '2023-07-07T06:00:59.470873+0000', 'last_active': '2023-07-07T06:00:59.480068+0000', 'last_peered': '2023-07-07T06:00:59.480068+0000', 'last_clean': '2023-07-07T06:00:35.066600+0000', 'last_became_active': '2023-07-07T06:00:59.470873+0000', 'last_became_peered': '2023-07-07T06:00:59.470873+0000', 'last_unstale': '2023-07-07T06:00:59.480068+0000', 'last_undegraded': '2023-07-07T06:00:59.462245+0000', 'last_fullsized': '2023-07-07T06:00:59.461717+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'objects_scrubbed': 0, 'log_size': 123, 'log_dups_size': 0, 'ondisk_log_size': 123, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 96893477, 'num_objects': 34, 'num_object_clones': 0, 'num_object_copies': 68, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 34, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 107, 'num_read_kb': 31202, 'num_write': 158, 'num_write_kb': 47074, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 34}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}

2023-07-07T17:34:03.810 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:04:48.624 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean
, 2023-07-07T06:04:48.624 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "14'97", 'reported_seq': 155, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.469971+0000', 'last_change': '2023-07-07T06:00:59.469575+0000', 'last_active': '2023-07-07T06:00:59.469971+0000', 'last_peered': '2023-07-07T06:00:59.469971+0000', 'last_clean': '2023-07-07T06:00:23.369029+0000', 'last_became_active': '2023-07-07T06:00:59.469575+0000', 'last_became_peered': '2023-07-07T06:00:59.469575+0000', 'last_unstale': '2023-07-07T06:00:59.469971+0000', 'last_undegraded': '2023-07-07T06:00:59.465049+0000', 'last_fullsized': '2023-07-07T06:00:59.464546+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []}
, 2023-07-07T06:04:48.625 INFO:tasks.ceph.ceph_manager.ceph:PG 3.2 is not active+clean

2023-07-07T17:34:03.811 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:04:57.681 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.468569+0000', 'last_change': '2023-07-07T06:00:59.468122+0000', 'last_active': '2023-07-07T06:00:59.468569+0000', 'last_peered': '2023-07-07T06:00:59.468569+0000', 'last_clean': '2023-07-07T06:00:25.415275+0000', 'last_became_active': '2023-07-07T06:00:59.468122+0000', 'last_became_peered': '2023-07-07T06:00:59.468122+0000', 'last_unstale': '2023-07-07T06:00:59.468569+0000', 'last_undegraded': '2023-07-07T06:00:59.468569+0000', 'last_fullsized': '2023-07-07T06:00:59.457597+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:04:57.682 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean
, 2023-07-07T06:04:57.682 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "14'97", 'reported_seq': 155, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.469971+0000', 'last_change': '2023-07-07T06:00:59.469575+0000', 'last_active': '2023-07-07T06:00:59.469971+0000', 'last_peered': '2023-07-07T06:00:59.469971+0000', 'last_clean': '2023-07-07T06:00:23.369029+0000', 'last_became_active': '2023-07-07T06:00:59.469575+0000', 'last_became_peered': '2023-07-07T06:00:59.469575+0000', 'last_unstale': '2023-07-07T06:00:59.469971+0000', 'last_undegraded': '2023-07-07T06:00:59.465049+0000', 'last_fullsized': '2023-07-07T06:00:59.464546+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []}

2023-07-07T17:34:03.813 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:05:06.737 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.468569+0000', 'last_change': '2023-07-07T06:00:59.468122+0000', 'last_active': '2023-07-07T06:00:59.468569+0000', 'last_peered': '2023-07-07T06:00:59.468569+0000', 'last_clean': '2023-07-07T06:00:25.415275+0000', 'last_became_active': '2023-07-07T06:00:59.468122+0000', 'last_became_peered': '2023-07-07T06:00:59.468122+0000', 'last_unstale': '2023-07-07T06:00:59.468569+0000', 'last_undegraded': '2023-07-07T06:00:59.468569+0000', 'last_fullsized': '2023-07-07T06:00:59.457597+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:05:06.738 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean
, 2023-07-07T06:05:06.738 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "14'97", 'reported_seq': 155, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.469971+0000', 'last_change': '2023-07-07T06:00:59.469575+0000', 'last_active': '2023-07-07T06:00:59.469971+0000', 'last_peered': '2023-07-07T06:00:59.469971+0000', 'last_clean': '2023-07-07T06:00:23.369029+0000', 'last_became_active': '2023-07-07T06:00:59.469575+0000', 'last_became_peered': '2023-07-07T06:00:59.469575+0000', 'last_unstale': '2023-07-07T06:00:59.469971+0000', 'last_undegraded': '2023-07-07T06:00:59.465049+0000', 'last_fullsized': '2023-07-07T06:00:59.464546+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:06.296978+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 103, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []}

2023-07-07T17:34:03.814 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:05:15.804 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean
, 2023-07-07T06:05:15.805 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.468569+0000', 'last_change': '2023-07-07T06:00:59.468122+0000', 'last_active': '2023-07-07T06:00:59.468569+0000', 'last_peered': '2023-07-07T06:00:59.468569+0000', 'last_clean': '2023-07-07T06:00:25.415275+0000', 'last_became_active': '2023-07-07T06:00:59.468122+0000', 'last_became_peered': '2023-07-07T06:00:59.468122+0000', 'last_unstale': '2023-07-07T06:00:59.468569+0000', 'last_undegraded': '2023-07-07T06:00:59.468569+0000', 'last_fullsized': '2023-07-07T06:00:59.457597+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}
, 2023-07-07T06:05:15.805 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean

2023-07-07T17:34:03.816 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:05:24.890 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.3', 'version': "21'107", 'reported_seq': 156, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.478255+0000', 'last_change': '2023-07-07T06:00:59.464604+0000', 'last_active': '2023-07-07T06:00:59.478255+0000', 'last_peered': '2023-07-07T06:00:59.478255+0000', 'last_clean': '2023-07-07T06:00:33.546473+0000', 'last_became_active': '2023-07-07T06:00:59.464604+0000', 'last_became_peered': '2023-07-07T06:00:59.464604+0000', 'last_unstale': '2023-07-07T06:00:59.478255+0000', 'last_undegraded': '2023-07-07T06:00:59.458856+0000', 'last_fullsized': '2023-07-07T06:00:59.458231+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'objects_scrubbed': 0, 'log_size': 107, 'log_dups_size': 0, 'ondisk_log_size': 107, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 82623806, 'num_objects': 29, 'num_object_clones': 0, 'num_object_copies': 58, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 29, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 61, 'num_read_kb': 31, 'num_write': 138, 'num_write_kb': 39292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 29}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []}
, 2023-07-07T06:05:24.891 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean
, 2023-07-07T06:05:24.891 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.468569+0000', 'last_change': '2023-07-07T06:00:59.468122+0000', 'last_active': '2023-07-07T06:00:59.468569+0000', 'last_peered': '2023-07-07T06:00:59.468569+0000', 'last_clean': '2023-07-07T06:00:25.415275+0000', 'last_became_active': '2023-07-07T06:00:59.468122+0000', 'last_became_peered': '2023-07-07T06:00:59.468122+0000', 'last_unstale': '2023-07-07T06:00:59.468569+0000', 'last_undegraded': '2023-07-07T06:00:59.468569+0000', 'last_fullsized': '2023-07-07T06:00:59.457597+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}

2023-07-07T17:34:03.817 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:05:33.966 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.3', 'version': "21'107", 'reported_seq': 156, 'reported_epoch': 21, 'state': 'active+undersized+degraded', 'last_fresh': '2023-07-07T06:00:59.478255+0000', 'last_change': '2023-07-07T06:00:59.464604+0000', 'last_active': '2023-07-07T06:00:59.478255+0000', 'last_peered': '2023-07-07T06:00:59.478255+0000', 'last_clean': '2023-07-07T06:00:33.546473+0000', 'last_became_active': '2023-07-07T06:00:59.464604+0000', 'last_became_peered': '2023-07-07T06:00:59.464604+0000', 'last_unstale': '2023-07-07T06:00:59.478255+0000', 'last_undegraded': '2023-07-07T06:00:59.458856+0000', 'last_fullsized': '2023-07-07T06:00:59.458231+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:19.370557+0000', 'objects_scrubbed': 0, 'log_size': 107, 'log_dups_size': 0, 'ondisk_log_size': 107, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 82623806, 'num_objects': 29, 'num_object_clones': 0, 'num_object_copies': 58, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 29, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 61, 'num_read_kb': 31, 'num_write': 138, 'num_write_kb': 39292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 29}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []}
, 2023-07-07T06:05:33.967 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean
, 2023-07-07T06:05:33.967 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 21, 'state': 'active+undersized', 'last_fresh': '2023-07-07T06:00:59.468569+0000', 'last_change': '2023-07-07T06:00:59.468122+0000', 'last_active': '2023-07-07T06:00:59.468569+0000', 'last_peered': '2023-07-07T06:00:59.468569+0000', 'last_clean': '2023-07-07T06:00:25.415275+0000', 'last_became_active': '2023-07-07T06:00:59.468122+0000', 'last_became_peered': '2023-07-07T06:00:59.468122+0000', 'last_unstale': '2023-07-07T06:00:59.468569+0000', 'last_undegraded': '2023-07-07T06:00:59.468569+0000', 'last_fullsized': '2023-07-07T06:00:59.457597+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'last_clean_scrub_stamp': '2023-07-07T06:00:08.305657+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []}

2023-07-07T17:34:03.817 WARNING:scrape:Ignoring malparsed backtrace:   File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception
,   File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise
,     raise value.with_traceback(tb)

2023-07-07T17:34:03.818 WARNING:scrape:Ignoring malparsed backtrace: Traceback (most recent call last):
,   File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/daemon/state.py", line 139, in stop
,     run.wait([self.proc], timeout=timeout)

2023-07-07T17:34:03.818 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 479, in wait
,     proc.wait()
,   File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait

2023-07-07T17:34:03.819 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait
,     self._raise_for_status()
,   File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status

2023-07-07T17:34:03.819 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.115332.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116271.log
,  49.6% -- replaced with /var/log/ceph/ceph-client.admin.115332.log.gz
, /var/log/ceph/ceph-client.admin.121237.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114695.log

2023-07-07T17:34:03.820 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.123891.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120395.log
,  49.2% -- replaced with /var/log/ceph/ceph-client.admin.123891.log.gz
, /var/log/ceph/ceph-client.admin.124136.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121678.log

2023-07-07T17:34:03.820 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.123695.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120698.log
,  49.6% -- replaced with /var/log/ceph/ceph-client.admin.123695.log.gz
, /var/log/ceph/ceph-client.admin.117300.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113421.log

2023-07-07T17:34:03.821 WARNING:scrape:Ignoring malparsed backtrace:  50.0% -- replaced with /var/log/ceph/ceph-client.admin.114303.log.gz
, gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122217.log
, /var/log/ceph/ceph-client.admin.123001.log:	 49.6% -- replaced with /var/log/ceph/ceph-client.admin.123001.log.gz/var/log/ceph/ceph-client.admin.111201.log:

2023-07-07T17:34:03.821 WARNING:scrape:Ignoring malparsed backtrace:  49.6% -- replaced with /var/log/ceph/ceph-client.admin.117643.log.gz
, /var/log/ceph/ceph-client.admin.122119.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123940.log
,  49.6% -- replaced with /var/log/ceph/ceph-client.admin.122119.log.gz

2023-07-07T17:34:03.822 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.121531.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118288.log
,  50.0% -- replaced with /var/log/ceph/ceph-client.admin.121531.log.gz
, /var/log/ceph/ceph-client.admin.118974.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115781.log

2023-07-07T17:34:03.822 WARNING:scrape:Ignoring malparsed backtrace: Traceback (most recent call last):
,   File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run_tasks.py", line 192, in run_tasks
,     suppress = manager.__exit__(*exc_info)

2023-07-07T17:34:03.822 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:07:45.041 DEBUG:teuthology.orchestra.run.smithi027:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz
, 2023-07-07T06:07:45.063 DEBUG:teuthology.orchestra.run.smithi167:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz
, 2023-07-07T06:07:45.145 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo

2023-07-07T17:34:06.417 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.0.0-4717-g6e5383fd (6e5383fdd6d402108195ddbbefb66fca2cebcf96) reef (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7fa928f2ced4]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x61b) [0x560d0fe70535]

2023-07-07T17:34:06.469 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:00:53.399 INFO:teuthology.orchestra.run.smithi018.stdout:
, 2023-07-07T06:00:53.400 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":27,"fsid":"c56c1d67-6b97-42b4-961e-90cf7e842363","created":"2023-07-07T05:59:50.010948+0000","modified":"2023-07-07T06:00:53.026768+0000","last_up_change":"2023-07-07T06:00:39.977315+0000","last_in_change":"2023-07-07T05:59:52.305625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-07T06:00:00.259409+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-07T06:00:00.646247+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":0.75,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2023-07-07T06:00:07.792277+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":1.5099999904632568,"score_stable":1.5099999904632568,"optimal_score":0.75,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b1a51f9-1515-4a38-b5dd-30e7dab92edd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6803","nonce":1830563861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":1830563861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6807","nonce":1830563861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":1830563861}]},"public_addr":"172.21.15.18:6803/1830563861","cluster_addr":"172.21.15.18:6802/1830563861","heartbeat_back_addr":"172.21.15.18:6807/1830563861","heartbeat_front_addr":"172.21.15.18:6806/1830563861","state":["exists"]},{"osd":1,"uuid":"e16c3d27-989a-43a5-b811-7a8029956383","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6801","nonce":45406582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":45406582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6805","nonce":45406582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":45406582}]},"public_addr":"172.21.15.18:6801/45406582","cluster_addr":"172.21.15.18:6800/45406582","heartbeat_back_addr":"172.21.15.18:6805/45406582","heartbeat_front_addr":"172.21.15.18:6804/45406582","state":["exists","up"]},{"osd":2,"uuid":"d8a24ad0-d28f-4794-bc1e-14191a09cafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2277875865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":2277875865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2277875865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":2277875865}]},"public_addr":"172.21.15.44:6804/2277875865","cluster_addr":"172.21.15.44:6805/2277875865","heartbeat_back_addr":"172.21.15.44:6808/2277875865","heartbeat_front_addr":"172.21.15.44:6809/2277875865","state":["exists","up"]},{"osd":3,"uuid":"fc7f5fb6-136c-406b-8f74-17ffe6776ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6803","nonce":3912049469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3912049469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3912049469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6807","nonce":3912049469}]},"public_addr":"172.21.15.44:6803/3912049469","cluster_addr":"172.21.15.44:6802/3912049469","heartbeat_back_addr":"172.21.15.44:6806/3912049469","heartbeat_front_addr":"172.21.15.44:6807/3912049469","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2023-07-07T06:00:39.977315+0000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2023-07-07T06:00:56.415 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json

2023-07-07T17:34:06.470 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:01:54.103 INFO:teuthology.orchestra.run.smithi018.stdout:
, 2023-07-07T06:01:54.103 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":27,"fsid":"c56c1d67-6b97-42b4-961e-90cf7e842363","created":"2023-07-07T05:59:50.010948+0000","modified":"2023-07-07T06:00:53.026768+0000","last_up_change":"2023-07-07T06:00:39.977315+0000","last_in_change":"2023-07-07T05:59:52.305625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-07T06:00:00.259409+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-07T06:00:00.646247+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":0.75,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2023-07-07T06:00:07.792277+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":1.5099999904632568,"score_stable":1.5099999904632568,"optimal_score":0.75,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b1a51f9-1515-4a38-b5dd-30e7dab92edd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6803","nonce":1830563861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":1830563861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6807","nonce":1830563861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":1830563861}]},"public_addr":"172.21.15.18:6803/1830563861","cluster_addr":"172.21.15.18:6802/1830563861","heartbeat_back_addr":"172.21.15.18:6807/1830563861","heartbeat_front_addr":"172.21.15.18:6806/1830563861","state":["exists"]},{"osd":1,"uuid":"e16c3d27-989a-43a5-b811-7a8029956383","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6801","nonce":45406582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":45406582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6805","nonce":45406582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":45406582}]},"public_addr":"172.21.15.18:6801/45406582","cluster_addr":"172.21.15.18:6800/45406582","heartbeat_back_addr":"172.21.15.18:6805/45406582","heartbeat_front_addr":"172.21.15.18:6804/45406582","state":["exists","up"]},{"osd":2,"uuid":"d8a24ad0-d28f-4794-bc1e-14191a09cafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2277875865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":2277875865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2277875865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":2277875865}]},"public_addr":"172.21.15.44:6804/2277875865","cluster_addr":"172.21.15.44:6805/2277875865","heartbeat_back_addr":"172.21.15.44:6808/2277875865","heartbeat_front_addr":"172.21.15.44:6809/2277875865","state":["exists","up"]},{"osd":3,"uuid":"fc7f5fb6-136c-406b-8f74-17ffe6776ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6803","nonce":3912049469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3912049469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3912049469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6807","nonce":3912049469}]},"public_addr":"172.21.15.44:6803/3912049469","cluster_addr":"172.21.15.44:6802/3912049469","heartbeat_back_addr":"172.21.15.44:6806/3912049469","heartbeat_front_addr":"172.21.15.44:6807/3912049469","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2023-07-07T06:00:39.977315+0000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2023-07-07T06:01:57.118 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json

2023-07-07T17:34:06.471 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:02:54.627 INFO:teuthology.orchestra.run.smithi018.stdout:
, 2023-07-07T06:02:54.627 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":27,"fsid":"c56c1d67-6b97-42b4-961e-90cf7e842363","created":"2023-07-07T05:59:50.010948+0000","modified":"2023-07-07T06:00:53.026768+0000","last_up_change":"2023-07-07T06:00:39.977315+0000","last_in_change":"2023-07-07T05:59:52.305625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-07T06:00:00.259409+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-07T06:00:00.646247+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":0.75,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2023-07-07T06:00:07.792277+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":1.5099999904632568,"score_stable":1.5099999904632568,"optimal_score":0.75,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b1a51f9-1515-4a38-b5dd-30e7dab92edd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6803","nonce":1830563861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":1830563861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6807","nonce":1830563861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":1830563861}]},"public_addr":"172.21.15.18:6803/1830563861","cluster_addr":"172.21.15.18:6802/1830563861","heartbeat_back_addr":"172.21.15.18:6807/1830563861","heartbeat_front_addr":"172.21.15.18:6806/1830563861","state":["exists"]},{"osd":1,"uuid":"e16c3d27-989a-43a5-b811-7a8029956383","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6801","nonce":45406582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":45406582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6805","nonce":45406582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":45406582}]},"public_addr":"172.21.15.18:6801/45406582","cluster_addr":"172.21.15.18:6800/45406582","heartbeat_back_addr":"172.21.15.18:6805/45406582","heartbeat_front_addr":"172.21.15.18:6804/45406582","state":["exists","up"]},{"osd":2,"uuid":"d8a24ad0-d28f-4794-bc1e-14191a09cafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2277875865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":2277875865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2277875865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":2277875865}]},"public_addr":"172.21.15.44:6804/2277875865","cluster_addr":"172.21.15.44:6805/2277875865","heartbeat_back_addr":"172.21.15.44:6808/2277875865","heartbeat_front_addr":"172.21.15.44:6809/2277875865","state":["exists","up"]},{"osd":3,"uuid":"fc7f5fb6-136c-406b-8f74-17ffe6776ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6803","nonce":3912049469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3912049469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3912049469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6807","nonce":3912049469}]},"public_addr":"172.21.15.44:6803/3912049469","cluster_addr":"172.21.15.44:6802/3912049469","heartbeat_back_addr":"172.21.15.44:6806/3912049469","heartbeat_front_addr":"172.21.15.44:6807/3912049469","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2023-07-07T06:00:39.977315+0000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2023-07-07T06:02:57.642 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json

2023-07-07T17:34:06.472 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:03:55.356 INFO:teuthology.orchestra.run.smithi018.stdout:
, 2023-07-07T06:03:55.356 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":27,"fsid":"c56c1d67-6b97-42b4-961e-90cf7e842363","created":"2023-07-07T05:59:50.010948+0000","modified":"2023-07-07T06:00:53.026768+0000","last_up_change":"2023-07-07T06:00:39.977315+0000","last_in_change":"2023-07-07T05:59:52.305625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-07T06:00:00.259409+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-07T06:00:00.646247+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":0.75,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2023-07-07T06:00:07.792277+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":1.5099999904632568,"score_stable":1.5099999904632568,"optimal_score":0.75,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b1a51f9-1515-4a38-b5dd-30e7dab92edd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6803","nonce":1830563861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":1830563861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6807","nonce":1830563861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":1830563861}]},"public_addr":"172.21.15.18:6803/1830563861","cluster_addr":"172.21.15.18:6802/1830563861","heartbeat_back_addr":"172.21.15.18:6807/1830563861","heartbeat_front_addr":"172.21.15.18:6806/1830563861","state":["exists"]},{"osd":1,"uuid":"e16c3d27-989a-43a5-b811-7a8029956383","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6801","nonce":45406582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":45406582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6805","nonce":45406582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":45406582}]},"public_addr":"172.21.15.18:6801/45406582","cluster_addr":"172.21.15.18:6800/45406582","heartbeat_back_addr":"172.21.15.18:6805/45406582","heartbeat_front_addr":"172.21.15.18:6804/45406582","state":["exists","up"]},{"osd":2,"uuid":"d8a24ad0-d28f-4794-bc1e-14191a09cafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2277875865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":2277875865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2277875865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":2277875865}]},"public_addr":"172.21.15.44:6804/2277875865","cluster_addr":"172.21.15.44:6805/2277875865","heartbeat_back_addr":"172.21.15.44:6808/2277875865","heartbeat_front_addr":"172.21.15.44:6809/2277875865","state":["exists","up"]},{"osd":3,"uuid":"fc7f5fb6-136c-406b-8f74-17ffe6776ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6803","nonce":3912049469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3912049469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3912049469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6807","nonce":3912049469}]},"public_addr":"172.21.15.44:6803/3912049469","cluster_addr":"172.21.15.44:6802/3912049469","heartbeat_back_addr":"172.21.15.44:6806/3912049469","heartbeat_front_addr":"172.21.15.44:6807/3912049469","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2023-07-07T06:00:39.977315+0000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2023-07-07T06:03:58.372 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json

2023-07-07T17:34:06.473 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:04:56.197 INFO:teuthology.orchestra.run.smithi018.stdout:
, 2023-07-07T06:04:56.197 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":27,"fsid":"c56c1d67-6b97-42b4-961e-90cf7e842363","created":"2023-07-07T05:59:50.010948+0000","modified":"2023-07-07T06:00:53.026768+0000","last_up_change":"2023-07-07T06:00:39.977315+0000","last_in_change":"2023-07-07T05:59:52.305625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-07T06:00:00.259409+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-07T06:00:00.646247+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":0.75,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2023-07-07T06:00:07.792277+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":1.5099999904632568,"score_stable":1.5099999904632568,"optimal_score":0.75,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b1a51f9-1515-4a38-b5dd-30e7dab92edd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6803","nonce":1830563861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":1830563861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6807","nonce":1830563861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":1830563861}]},"public_addr":"172.21.15.18:6803/1830563861","cluster_addr":"172.21.15.18:6802/1830563861","heartbeat_back_addr":"172.21.15.18:6807/1830563861","heartbeat_front_addr":"172.21.15.18:6806/1830563861","state":["exists"]},{"osd":1,"uuid":"e16c3d27-989a-43a5-b811-7a8029956383","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6801","nonce":45406582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":45406582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6805","nonce":45406582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":45406582}]},"public_addr":"172.21.15.18:6801/45406582","cluster_addr":"172.21.15.18:6800/45406582","heartbeat_back_addr":"172.21.15.18:6805/45406582","heartbeat_front_addr":"172.21.15.18:6804/45406582","state":["exists","up"]},{"osd":2,"uuid":"d8a24ad0-d28f-4794-bc1e-14191a09cafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2277875865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":2277875865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2277875865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":2277875865}]},"public_addr":"172.21.15.44:6804/2277875865","cluster_addr":"172.21.15.44:6805/2277875865","heartbeat_back_addr":"172.21.15.44:6808/2277875865","heartbeat_front_addr":"172.21.15.44:6809/2277875865","state":["exists","up"]},{"osd":3,"uuid":"fc7f5fb6-136c-406b-8f74-17ffe6776ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6803","nonce":3912049469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3912049469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3912049469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6807","nonce":3912049469}]},"public_addr":"172.21.15.44:6803/3912049469","cluster_addr":"172.21.15.44:6802/3912049469","heartbeat_back_addr":"172.21.15.44:6806/3912049469","heartbeat_front_addr":"172.21.15.44:6807/3912049469","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2023-07-07T06:00:39.977315+0000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2023-07-07T06:04:59.220 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json

2023-07-07T17:34:06.473 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status
,     raise CommandFailedError(
, teuthology.exceptions.CommandFailedError: Command failed on smithi018 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json'

2023-07-07T17:34:06.474 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:57:31.882 DEBUG:tasks.ceph.mgr.x:waiting for process to exit
, 2023-07-07T06:57:31.882 INFO:teuthology.orchestra.run:waiting for 300
, 2023-07-07T06:57:31.883 INFO:tasks.ceph.mgr.x:Stopped

2023-07-07T17:34:06.474 WARNING:scrape:Ignoring malparsed backtrace:  49.6% -- replaced with /var/log/ceph/ceph-client.admin.111618.log.gz
, /var/log/ceph/ceph-client.admin.112206.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114435.log
,  49.6% -- replaced with /var/log/ceph/ceph-client.admin.112206.log.gz

2023-07-07T17:34:06.475 WARNING:scrape:Ignoring malparsed backtrace:  50.0% -- replaced with /var/log/ceph/ceph-client.admin.109033.log.gz
, /var/log/ceph/ceph-client.admin.110421.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113692.log
,  49.6% -- replaced with /var/log/ceph/ceph-client.admin.110421.log.gz

2023-07-07T17:34:06.475 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.110715.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110519.log
,  47.6% -- replaced with /var/log/ceph/ceph-client.admin.110715.log.gz
, /var/log/ceph/ceph-client.admin.114582.log:	gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log

2023-07-07T17:34:06.476 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:57:34.156 DEBUG:teuthology.orchestra.run.smithi018:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart
, 2023-07-07T06:57:34.158 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart
, Redirecting to /bin/systemctl restart rsyslog.service

2023-07-07T17:34:06.476 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:57:45.506 INFO:teuthology.orchestra.run.smithi018.stdout:   393224      4 drwxr-xr-x   3  ubuntu   ubuntu       4096 Jul  7 06:57 /home/ubuntu/cephtest
, 2023-07-07T06:57:45.506 INFO:teuthology.orchestra.run.smithi018.stdout:   393225      4 drwxr-xr-x   5  ubuntu   ubuntu       4096 Jul  7 06:57 /home/ubuntu/cephtest/archive
, 2023-07-07T06:57:45.506 INFO:teuthology.orchestra.run.smithi018.stdout:   393226      4 drwxr-xr-x   2  ubuntu   ubuntu       4096 Jul  7 06:00 /home/ubuntu/cephtest/archive/coredump

2023-07-07T17:34:06.489 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.0.0-4717-g6e5383fd (6e5383fdd6d402108195ddbbefb66fca2cebcf96) reef (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7fa928f2ced4]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x61b) [0x560d0fe70535]

2023-07-07T17:34:06.490 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:00:53.399 INFO:teuthology.orchestra.run.smithi018.stdout:
, 2023-07-07T06:00:53.400 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":27,"fsid":"c56c1d67-6b97-42b4-961e-90cf7e842363","created":"2023-07-07T05:59:50.010948+0000","modified":"2023-07-07T06:00:53.026768+0000","last_up_change":"2023-07-07T06:00:39.977315+0000","last_in_change":"2023-07-07T05:59:52.305625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-07T06:00:00.259409+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-07T06:00:00.646247+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":0.75,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2023-07-07T06:00:07.792277+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":1.5099999904632568,"score_stable":1.5099999904632568,"optimal_score":0.75,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b1a51f9-1515-4a38-b5dd-30e7dab92edd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6803","nonce":1830563861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":1830563861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6807","nonce":1830563861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":1830563861}]},"public_addr":"172.21.15.18:6803/1830563861","cluster_addr":"172.21.15.18:6802/1830563861","heartbeat_back_addr":"172.21.15.18:6807/1830563861","heartbeat_front_addr":"172.21.15.18:6806/1830563861","state":["exists"]},{"osd":1,"uuid":"e16c3d27-989a-43a5-b811-7a8029956383","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6801","nonce":45406582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":45406582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6805","nonce":45406582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":45406582}]},"public_addr":"172.21.15.18:6801/45406582","cluster_addr":"172.21.15.18:6800/45406582","heartbeat_back_addr":"172.21.15.18:6805/45406582","heartbeat_front_addr":"172.21.15.18:6804/45406582","state":["exists","up"]},{"osd":2,"uuid":"d8a24ad0-d28f-4794-bc1e-14191a09cafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2277875865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":2277875865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2277875865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":2277875865}]},"public_addr":"172.21.15.44:6804/2277875865","cluster_addr":"172.21.15.44:6805/2277875865","heartbeat_back_addr":"172.21.15.44:6808/2277875865","heartbeat_front_addr":"172.21.15.44:6809/2277875865","state":["exists","up"]},{"osd":3,"uuid":"fc7f5fb6-136c-406b-8f74-17ffe6776ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6803","nonce":3912049469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3912049469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3912049469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6807","nonce":3912049469}]},"public_addr":"172.21.15.44:6803/3912049469","cluster_addr":"172.21.15.44:6802/3912049469","heartbeat_back_addr":"172.21.15.44:6806/3912049469","heartbeat_front_addr":"172.21.15.44:6807/3912049469","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2023-07-07T06:00:39.977315+0000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2023-07-07T06:00:56.415 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json

2023-07-07T17:34:06.491 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:01:54.103 INFO:teuthology.orchestra.run.smithi018.stdout:
, 2023-07-07T06:01:54.103 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":27,"fsid":"c56c1d67-6b97-42b4-961e-90cf7e842363","created":"2023-07-07T05:59:50.010948+0000","modified":"2023-07-07T06:00:53.026768+0000","last_up_change":"2023-07-07T06:00:39.977315+0000","last_in_change":"2023-07-07T05:59:52.305625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-07T06:00:00.259409+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-07T06:00:00.646247+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":0.75,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2023-07-07T06:00:07.792277+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":1.5099999904632568,"score_stable":1.5099999904632568,"optimal_score":0.75,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b1a51f9-1515-4a38-b5dd-30e7dab92edd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6803","nonce":1830563861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":1830563861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6807","nonce":1830563861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":1830563861}]},"public_addr":"172.21.15.18:6803/1830563861","cluster_addr":"172.21.15.18:6802/1830563861","heartbeat_back_addr":"172.21.15.18:6807/1830563861","heartbeat_front_addr":"172.21.15.18:6806/1830563861","state":["exists"]},{"osd":1,"uuid":"e16c3d27-989a-43a5-b811-7a8029956383","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6801","nonce":45406582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":45406582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6805","nonce":45406582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":45406582}]},"public_addr":"172.21.15.18:6801/45406582","cluster_addr":"172.21.15.18:6800/45406582","heartbeat_back_addr":"172.21.15.18:6805/45406582","heartbeat_front_addr":"172.21.15.18:6804/45406582","state":["exists","up"]},{"osd":2,"uuid":"d8a24ad0-d28f-4794-bc1e-14191a09cafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2277875865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":2277875865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2277875865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":2277875865}]},"public_addr":"172.21.15.44:6804/2277875865","cluster_addr":"172.21.15.44:6805/2277875865","heartbeat_back_addr":"172.21.15.44:6808/2277875865","heartbeat_front_addr":"172.21.15.44:6809/2277875865","state":["exists","up"]},{"osd":3,"uuid":"fc7f5fb6-136c-406b-8f74-17ffe6776ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6803","nonce":3912049469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3912049469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3912049469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6807","nonce":3912049469}]},"public_addr":"172.21.15.44:6803/3912049469","cluster_addr":"172.21.15.44:6802/3912049469","heartbeat_back_addr":"172.21.15.44:6806/3912049469","heartbeat_front_addr":"172.21.15.44:6807/3912049469","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2023-07-07T06:00:39.977315+0000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2023-07-07T06:01:57.118 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json

2023-07-07T17:34:06.492 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:02:54.627 INFO:teuthology.orchestra.run.smithi018.stdout:
, 2023-07-07T06:02:54.627 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":27,"fsid":"c56c1d67-6b97-42b4-961e-90cf7e842363","created":"2023-07-07T05:59:50.010948+0000","modified":"2023-07-07T06:00:53.026768+0000","last_up_change":"2023-07-07T06:00:39.977315+0000","last_in_change":"2023-07-07T05:59:52.305625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-07T06:00:00.259409+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-07T06:00:00.646247+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":0.75,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2023-07-07T06:00:07.792277+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":1.5099999904632568,"score_stable":1.5099999904632568,"optimal_score":0.75,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b1a51f9-1515-4a38-b5dd-30e7dab92edd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6803","nonce":1830563861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":1830563861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6807","nonce":1830563861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":1830563861}]},"public_addr":"172.21.15.18:6803/1830563861","cluster_addr":"172.21.15.18:6802/1830563861","heartbeat_back_addr":"172.21.15.18:6807/1830563861","heartbeat_front_addr":"172.21.15.18:6806/1830563861","state":["exists"]},{"osd":1,"uuid":"e16c3d27-989a-43a5-b811-7a8029956383","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6801","nonce":45406582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":45406582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6805","nonce":45406582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":45406582}]},"public_addr":"172.21.15.18:6801/45406582","cluster_addr":"172.21.15.18:6800/45406582","heartbeat_back_addr":"172.21.15.18:6805/45406582","heartbeat_front_addr":"172.21.15.18:6804/45406582","state":["exists","up"]},{"osd":2,"uuid":"d8a24ad0-d28f-4794-bc1e-14191a09cafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2277875865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":2277875865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2277875865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":2277875865}]},"public_addr":"172.21.15.44:6804/2277875865","cluster_addr":"172.21.15.44:6805/2277875865","heartbeat_back_addr":"172.21.15.44:6808/2277875865","heartbeat_front_addr":"172.21.15.44:6809/2277875865","state":["exists","up"]},{"osd":3,"uuid":"fc7f5fb6-136c-406b-8f74-17ffe6776ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6803","nonce":3912049469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3912049469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3912049469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6807","nonce":3912049469}]},"public_addr":"172.21.15.44:6803/3912049469","cluster_addr":"172.21.15.44:6802/3912049469","heartbeat_back_addr":"172.21.15.44:6806/3912049469","heartbeat_front_addr":"172.21.15.44:6807/3912049469","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2023-07-07T06:00:39.977315+0000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2023-07-07T06:02:57.642 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json

2023-07-07T17:34:06.493 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:03:55.356 INFO:teuthology.orchestra.run.smithi018.stdout:
, 2023-07-07T06:03:55.356 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":27,"fsid":"c56c1d67-6b97-42b4-961e-90cf7e842363","created":"2023-07-07T05:59:50.010948+0000","modified":"2023-07-07T06:00:53.026768+0000","last_up_change":"2023-07-07T06:00:39.977315+0000","last_in_change":"2023-07-07T05:59:52.305625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-07T06:00:00.259409+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-07T06:00:00.646247+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":0.75,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2023-07-07T06:00:07.792277+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":1.5099999904632568,"score_stable":1.5099999904632568,"optimal_score":0.75,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b1a51f9-1515-4a38-b5dd-30e7dab92edd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6803","nonce":1830563861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":1830563861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6807","nonce":1830563861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":1830563861}]},"public_addr":"172.21.15.18:6803/1830563861","cluster_addr":"172.21.15.18:6802/1830563861","heartbeat_back_addr":"172.21.15.18:6807/1830563861","heartbeat_front_addr":"172.21.15.18:6806/1830563861","state":["exists"]},{"osd":1,"uuid":"e16c3d27-989a-43a5-b811-7a8029956383","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6801","nonce":45406582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":45406582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6805","nonce":45406582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":45406582}]},"public_addr":"172.21.15.18:6801/45406582","cluster_addr":"172.21.15.18:6800/45406582","heartbeat_back_addr":"172.21.15.18:6805/45406582","heartbeat_front_addr":"172.21.15.18:6804/45406582","state":["exists","up"]},{"osd":2,"uuid":"d8a24ad0-d28f-4794-bc1e-14191a09cafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2277875865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":2277875865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2277875865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":2277875865}]},"public_addr":"172.21.15.44:6804/2277875865","cluster_addr":"172.21.15.44:6805/2277875865","heartbeat_back_addr":"172.21.15.44:6808/2277875865","heartbeat_front_addr":"172.21.15.44:6809/2277875865","state":["exists","up"]},{"osd":3,"uuid":"fc7f5fb6-136c-406b-8f74-17ffe6776ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6803","nonce":3912049469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3912049469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3912049469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6807","nonce":3912049469}]},"public_addr":"172.21.15.44:6803/3912049469","cluster_addr":"172.21.15.44:6802/3912049469","heartbeat_back_addr":"172.21.15.44:6806/3912049469","heartbeat_front_addr":"172.21.15.44:6807/3912049469","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2023-07-07T06:00:39.977315+0000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2023-07-07T06:03:58.372 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json

2023-07-07T17:34:06.493 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:04:56.197 INFO:teuthology.orchestra.run.smithi018.stdout:
, 2023-07-07T06:04:56.197 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":27,"fsid":"c56c1d67-6b97-42b4-961e-90cf7e842363","created":"2023-07-07T05:59:50.010948+0000","modified":"2023-07-07T06:00:53.026768+0000","last_up_change":"2023-07-07T06:00:39.977315+0000","last_in_change":"2023-07-07T05:59:52.305625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-07T06:00:00.259409+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-07T06:00:00.646247+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":0.75,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2023-07-07T06:00:07.792277+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":1.5099999904632568,"score_stable":1.5099999904632568,"optimal_score":0.75,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b1a51f9-1515-4a38-b5dd-30e7dab92edd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6803","nonce":1830563861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":1830563861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6807","nonce":1830563861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":1830563861}]},"public_addr":"172.21.15.18:6803/1830563861","cluster_addr":"172.21.15.18:6802/1830563861","heartbeat_back_addr":"172.21.15.18:6807/1830563861","heartbeat_front_addr":"172.21.15.18:6806/1830563861","state":["exists"]},{"osd":1,"uuid":"e16c3d27-989a-43a5-b811-7a8029956383","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6801","nonce":45406582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":45406582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6805","nonce":45406582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":45406582}]},"public_addr":"172.21.15.18:6801/45406582","cluster_addr":"172.21.15.18:6800/45406582","heartbeat_back_addr":"172.21.15.18:6805/45406582","heartbeat_front_addr":"172.21.15.18:6804/45406582","state":["exists","up"]},{"osd":2,"uuid":"d8a24ad0-d28f-4794-bc1e-14191a09cafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2277875865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":2277875865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2277875865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":2277875865}]},"public_addr":"172.21.15.44:6804/2277875865","cluster_addr":"172.21.15.44:6805/2277875865","heartbeat_back_addr":"172.21.15.44:6808/2277875865","heartbeat_front_addr":"172.21.15.44:6809/2277875865","state":["exists","up"]},{"osd":3,"uuid":"fc7f5fb6-136c-406b-8f74-17ffe6776ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6803","nonce":3912049469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3912049469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3912049469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6807","nonce":3912049469}]},"public_addr":"172.21.15.44:6803/3912049469","cluster_addr":"172.21.15.44:6802/3912049469","heartbeat_back_addr":"172.21.15.44:6806/3912049469","heartbeat_front_addr":"172.21.15.44:6807/3912049469","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2023-07-07T06:00:39.977315+0000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2023-07-07T06:04:59.220 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json

2023-07-07T17:34:06.494 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status
,     raise CommandFailedError(
, teuthology.exceptions.CommandFailedError: Command failed on smithi018 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json'

2023-07-07T17:34:06.495 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:57:31.882 DEBUG:tasks.ceph.mgr.x:waiting for process to exit
, 2023-07-07T06:57:31.882 INFO:teuthology.orchestra.run:waiting for 300
, 2023-07-07T06:57:31.883 INFO:tasks.ceph.mgr.x:Stopped

2023-07-07T17:34:06.495 WARNING:scrape:Ignoring malparsed backtrace:  49.6% -- replaced with /var/log/ceph/ceph-client.admin.111618.log.gz
, /var/log/ceph/ceph-client.admin.112206.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114435.log
,  49.6% -- replaced with /var/log/ceph/ceph-client.admin.112206.log.gz

2023-07-07T17:34:06.496 WARNING:scrape:Ignoring malparsed backtrace:  50.0% -- replaced with /var/log/ceph/ceph-client.admin.109033.log.gz
, /var/log/ceph/ceph-client.admin.110421.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113692.log
,  49.6% -- replaced with /var/log/ceph/ceph-client.admin.110421.log.gz

2023-07-07T17:34:06.496 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.110715.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110519.log
,  47.6% -- replaced with /var/log/ceph/ceph-client.admin.110715.log.gz
, /var/log/ceph/ceph-client.admin.114582.log:	gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log

2023-07-07T17:34:06.497 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:57:34.156 DEBUG:teuthology.orchestra.run.smithi018:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart
, 2023-07-07T06:57:34.158 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart
, Redirecting to /bin/systemctl restart rsyslog.service

2023-07-07T17:34:06.497 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:57:45.506 INFO:teuthology.orchestra.run.smithi018.stdout:   393224      4 drwxr-xr-x   3  ubuntu   ubuntu       4096 Jul  7 06:57 /home/ubuntu/cephtest
, 2023-07-07T06:57:45.506 INFO:teuthology.orchestra.run.smithi018.stdout:   393225      4 drwxr-xr-x   5  ubuntu   ubuntu       4096 Jul  7 06:57 /home/ubuntu/cephtest/archive
, 2023-07-07T06:57:45.506 INFO:teuthology.orchestra.run.smithi018.stdout:   393226      4 drwxr-xr-x   2  ubuntu   ubuntu       4096 Jul  7 06:00 /home/ubuntu/cephtest/archive/coredump

2023-07-07T17:34:06.508 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.0.0-4717-g6e5383fd (6e5383fdd6d402108195ddbbefb66fca2cebcf96) reef (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7fa928f2ced4]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x61b) [0x560d0fe70535]

2023-07-07T17:34:06.509 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:00:53.399 INFO:teuthology.orchestra.run.smithi018.stdout:
, 2023-07-07T06:00:53.400 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":27,"fsid":"c56c1d67-6b97-42b4-961e-90cf7e842363","created":"2023-07-07T05:59:50.010948+0000","modified":"2023-07-07T06:00:53.026768+0000","last_up_change":"2023-07-07T06:00:39.977315+0000","last_in_change":"2023-07-07T05:59:52.305625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-07T06:00:00.259409+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-07T06:00:00.646247+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":0.75,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2023-07-07T06:00:07.792277+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":1.5099999904632568,"score_stable":1.5099999904632568,"optimal_score":0.75,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b1a51f9-1515-4a38-b5dd-30e7dab92edd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6803","nonce":1830563861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":1830563861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6807","nonce":1830563861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":1830563861}]},"public_addr":"172.21.15.18:6803/1830563861","cluster_addr":"172.21.15.18:6802/1830563861","heartbeat_back_addr":"172.21.15.18:6807/1830563861","heartbeat_front_addr":"172.21.15.18:6806/1830563861","state":["exists"]},{"osd":1,"uuid":"e16c3d27-989a-43a5-b811-7a8029956383","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6801","nonce":45406582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":45406582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6805","nonce":45406582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":45406582}]},"public_addr":"172.21.15.18:6801/45406582","cluster_addr":"172.21.15.18:6800/45406582","heartbeat_back_addr":"172.21.15.18:6805/45406582","heartbeat_front_addr":"172.21.15.18:6804/45406582","state":["exists","up"]},{"osd":2,"uuid":"d8a24ad0-d28f-4794-bc1e-14191a09cafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2277875865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":2277875865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2277875865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":2277875865}]},"public_addr":"172.21.15.44:6804/2277875865","cluster_addr":"172.21.15.44:6805/2277875865","heartbeat_back_addr":"172.21.15.44:6808/2277875865","heartbeat_front_addr":"172.21.15.44:6809/2277875865","state":["exists","up"]},{"osd":3,"uuid":"fc7f5fb6-136c-406b-8f74-17ffe6776ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6803","nonce":3912049469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3912049469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3912049469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6807","nonce":3912049469}]},"public_addr":"172.21.15.44:6803/3912049469","cluster_addr":"172.21.15.44:6802/3912049469","heartbeat_back_addr":"172.21.15.44:6806/3912049469","heartbeat_front_addr":"172.21.15.44:6807/3912049469","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2023-07-07T06:00:39.977315+0000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2023-07-07T06:00:56.415 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json

2023-07-07T17:34:06.510 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:01:54.103 INFO:teuthology.orchestra.run.smithi018.stdout:
, 2023-07-07T06:01:54.103 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":27,"fsid":"c56c1d67-6b97-42b4-961e-90cf7e842363","created":"2023-07-07T05:59:50.010948+0000","modified":"2023-07-07T06:00:53.026768+0000","last_up_change":"2023-07-07T06:00:39.977315+0000","last_in_change":"2023-07-07T05:59:52.305625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-07T06:00:00.259409+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-07T06:00:00.646247+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":0.75,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2023-07-07T06:00:07.792277+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":1.5099999904632568,"score_stable":1.5099999904632568,"optimal_score":0.75,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b1a51f9-1515-4a38-b5dd-30e7dab92edd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6803","nonce":1830563861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":1830563861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6807","nonce":1830563861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":1830563861}]},"public_addr":"172.21.15.18:6803/1830563861","cluster_addr":"172.21.15.18:6802/1830563861","heartbeat_back_addr":"172.21.15.18:6807/1830563861","heartbeat_front_addr":"172.21.15.18:6806/1830563861","state":["exists"]},{"osd":1,"uuid":"e16c3d27-989a-43a5-b811-7a8029956383","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6801","nonce":45406582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":45406582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6805","nonce":45406582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":45406582}]},"public_addr":"172.21.15.18:6801/45406582","cluster_addr":"172.21.15.18:6800/45406582","heartbeat_back_addr":"172.21.15.18:6805/45406582","heartbeat_front_addr":"172.21.15.18:6804/45406582","state":["exists","up"]},{"osd":2,"uuid":"d8a24ad0-d28f-4794-bc1e-14191a09cafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2277875865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":2277875865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2277875865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":2277875865}]},"public_addr":"172.21.15.44:6804/2277875865","cluster_addr":"172.21.15.44:6805/2277875865","heartbeat_back_addr":"172.21.15.44:6808/2277875865","heartbeat_front_addr":"172.21.15.44:6809/2277875865","state":["exists","up"]},{"osd":3,"uuid":"fc7f5fb6-136c-406b-8f74-17ffe6776ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6803","nonce":3912049469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3912049469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3912049469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6807","nonce":3912049469}]},"public_addr":"172.21.15.44:6803/3912049469","cluster_addr":"172.21.15.44:6802/3912049469","heartbeat_back_addr":"172.21.15.44:6806/3912049469","heartbeat_front_addr":"172.21.15.44:6807/3912049469","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2023-07-07T06:00:39.977315+0000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2023-07-07T06:01:57.118 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json

2023-07-07T17:34:06.511 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:02:54.627 INFO:teuthology.orchestra.run.smithi018.stdout:
, 2023-07-07T06:02:54.627 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":27,"fsid":"c56c1d67-6b97-42b4-961e-90cf7e842363","created":"2023-07-07T05:59:50.010948+0000","modified":"2023-07-07T06:00:53.026768+0000","last_up_change":"2023-07-07T06:00:39.977315+0000","last_in_change":"2023-07-07T05:59:52.305625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-07T06:00:00.259409+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-07T06:00:00.646247+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":0.75,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2023-07-07T06:00:07.792277+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":1.5099999904632568,"score_stable":1.5099999904632568,"optimal_score":0.75,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b1a51f9-1515-4a38-b5dd-30e7dab92edd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6803","nonce":1830563861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":1830563861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6807","nonce":1830563861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":1830563861}]},"public_addr":"172.21.15.18:6803/1830563861","cluster_addr":"172.21.15.18:6802/1830563861","heartbeat_back_addr":"172.21.15.18:6807/1830563861","heartbeat_front_addr":"172.21.15.18:6806/1830563861","state":["exists"]},{"osd":1,"uuid":"e16c3d27-989a-43a5-b811-7a8029956383","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6801","nonce":45406582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":45406582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6805","nonce":45406582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":45406582}]},"public_addr":"172.21.15.18:6801/45406582","cluster_addr":"172.21.15.18:6800/45406582","heartbeat_back_addr":"172.21.15.18:6805/45406582","heartbeat_front_addr":"172.21.15.18:6804/45406582","state":["exists","up"]},{"osd":2,"uuid":"d8a24ad0-d28f-4794-bc1e-14191a09cafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2277875865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":2277875865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2277875865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":2277875865}]},"public_addr":"172.21.15.44:6804/2277875865","cluster_addr":"172.21.15.44:6805/2277875865","heartbeat_back_addr":"172.21.15.44:6808/2277875865","heartbeat_front_addr":"172.21.15.44:6809/2277875865","state":["exists","up"]},{"osd":3,"uuid":"fc7f5fb6-136c-406b-8f74-17ffe6776ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6803","nonce":3912049469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3912049469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3912049469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6807","nonce":3912049469}]},"public_addr":"172.21.15.44:6803/3912049469","cluster_addr":"172.21.15.44:6802/3912049469","heartbeat_back_addr":"172.21.15.44:6806/3912049469","heartbeat_front_addr":"172.21.15.44:6807/3912049469","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2023-07-07T06:00:39.977315+0000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2023-07-07T06:02:57.642 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json

2023-07-07T17:34:06.512 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:03:55.356 INFO:teuthology.orchestra.run.smithi018.stdout:
, 2023-07-07T06:03:55.356 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":27,"fsid":"c56c1d67-6b97-42b4-961e-90cf7e842363","created":"2023-07-07T05:59:50.010948+0000","modified":"2023-07-07T06:00:53.026768+0000","last_up_change":"2023-07-07T06:00:39.977315+0000","last_in_change":"2023-07-07T05:59:52.305625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-07T06:00:00.259409+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-07T06:00:00.646247+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":0.75,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2023-07-07T06:00:07.792277+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":1.5099999904632568,"score_stable":1.5099999904632568,"optimal_score":0.75,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b1a51f9-1515-4a38-b5dd-30e7dab92edd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6803","nonce":1830563861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":1830563861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6807","nonce":1830563861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":1830563861}]},"public_addr":"172.21.15.18:6803/1830563861","cluster_addr":"172.21.15.18:6802/1830563861","heartbeat_back_addr":"172.21.15.18:6807/1830563861","heartbeat_front_addr":"172.21.15.18:6806/1830563861","state":["exists"]},{"osd":1,"uuid":"e16c3d27-989a-43a5-b811-7a8029956383","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6801","nonce":45406582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":45406582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6805","nonce":45406582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":45406582}]},"public_addr":"172.21.15.18:6801/45406582","cluster_addr":"172.21.15.18:6800/45406582","heartbeat_back_addr":"172.21.15.18:6805/45406582","heartbeat_front_addr":"172.21.15.18:6804/45406582","state":["exists","up"]},{"osd":2,"uuid":"d8a24ad0-d28f-4794-bc1e-14191a09cafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2277875865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":2277875865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2277875865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":2277875865}]},"public_addr":"172.21.15.44:6804/2277875865","cluster_addr":"172.21.15.44:6805/2277875865","heartbeat_back_addr":"172.21.15.44:6808/2277875865","heartbeat_front_addr":"172.21.15.44:6809/2277875865","state":["exists","up"]},{"osd":3,"uuid":"fc7f5fb6-136c-406b-8f74-17ffe6776ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6803","nonce":3912049469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3912049469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3912049469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6807","nonce":3912049469}]},"public_addr":"172.21.15.44:6803/3912049469","cluster_addr":"172.21.15.44:6802/3912049469","heartbeat_back_addr":"172.21.15.44:6806/3912049469","heartbeat_front_addr":"172.21.15.44:6807/3912049469","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2023-07-07T06:00:39.977315+0000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2023-07-07T06:03:58.372 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json

2023-07-07T17:34:06.513 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:04:56.197 INFO:teuthology.orchestra.run.smithi018.stdout:
, 2023-07-07T06:04:56.197 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":27,"fsid":"c56c1d67-6b97-42b4-961e-90cf7e842363","created":"2023-07-07T05:59:50.010948+0000","modified":"2023-07-07T06:00:53.026768+0000","last_up_change":"2023-07-07T06:00:39.977315+0000","last_in_change":"2023-07-07T05:59:52.305625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-07T06:00:00.259409+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-07T06:00:00.646247+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":0.75,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2023-07-07T06:00:07.792277+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":1.5099999904632568,"score_stable":1.5099999904632568,"optimal_score":0.75,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b1a51f9-1515-4a38-b5dd-30e7dab92edd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6803","nonce":1830563861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":1830563861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6807","nonce":1830563861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":1830563861}]},"public_addr":"172.21.15.18:6803/1830563861","cluster_addr":"172.21.15.18:6802/1830563861","heartbeat_back_addr":"172.21.15.18:6807/1830563861","heartbeat_front_addr":"172.21.15.18:6806/1830563861","state":["exists"]},{"osd":1,"uuid":"e16c3d27-989a-43a5-b811-7a8029956383","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6801","nonce":45406582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":45406582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6805","nonce":45406582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":45406582}]},"public_addr":"172.21.15.18:6801/45406582","cluster_addr":"172.21.15.18:6800/45406582","heartbeat_back_addr":"172.21.15.18:6805/45406582","heartbeat_front_addr":"172.21.15.18:6804/45406582","state":["exists","up"]},{"osd":2,"uuid":"d8a24ad0-d28f-4794-bc1e-14191a09cafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2277875865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":2277875865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2277875865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":2277875865}]},"public_addr":"172.21.15.44:6804/2277875865","cluster_addr":"172.21.15.44:6805/2277875865","heartbeat_back_addr":"172.21.15.44:6808/2277875865","heartbeat_front_addr":"172.21.15.44:6809/2277875865","state":["exists","up"]},{"osd":3,"uuid":"fc7f5fb6-136c-406b-8f74-17ffe6776ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6803","nonce":3912049469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3912049469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3912049469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6807","nonce":3912049469}]},"public_addr":"172.21.15.44:6803/3912049469","cluster_addr":"172.21.15.44:6802/3912049469","heartbeat_back_addr":"172.21.15.44:6806/3912049469","heartbeat_front_addr":"172.21.15.44:6807/3912049469","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2023-07-07T06:00:39.977315+0000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2023-07-07T06:04:59.220 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json

2023-07-07T17:34:06.513 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status
,     raise CommandFailedError(
, teuthology.exceptions.CommandFailedError: Command failed on smithi018 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json'

2023-07-07T17:34:06.514 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:57:31.882 DEBUG:tasks.ceph.mgr.x:waiting for process to exit
, 2023-07-07T06:57:31.882 INFO:teuthology.orchestra.run:waiting for 300
, 2023-07-07T06:57:31.883 INFO:tasks.ceph.mgr.x:Stopped

2023-07-07T17:34:06.514 WARNING:scrape:Ignoring malparsed backtrace:  49.6% -- replaced with /var/log/ceph/ceph-client.admin.111618.log.gz
, /var/log/ceph/ceph-client.admin.112206.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114435.log
,  49.6% -- replaced with /var/log/ceph/ceph-client.admin.112206.log.gz

2023-07-07T17:34:06.515 WARNING:scrape:Ignoring malparsed backtrace:  50.0% -- replaced with /var/log/ceph/ceph-client.admin.109033.log.gz
, /var/log/ceph/ceph-client.admin.110421.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113692.log
,  49.6% -- replaced with /var/log/ceph/ceph-client.admin.110421.log.gz

2023-07-07T17:34:06.515 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.110715.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110519.log
,  47.6% -- replaced with /var/log/ceph/ceph-client.admin.110715.log.gz
, /var/log/ceph/ceph-client.admin.114582.log:	gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log

2023-07-07T17:34:06.516 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:57:34.156 DEBUG:teuthology.orchestra.run.smithi018:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart
, 2023-07-07T06:57:34.158 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart
, Redirecting to /bin/systemctl restart rsyslog.service

2023-07-07T17:34:06.516 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:57:45.506 INFO:teuthology.orchestra.run.smithi018.stdout:   393224      4 drwxr-xr-x   3  ubuntu   ubuntu       4096 Jul  7 06:57 /home/ubuntu/cephtest
, 2023-07-07T06:57:45.506 INFO:teuthology.orchestra.run.smithi018.stdout:   393225      4 drwxr-xr-x   5  ubuntu   ubuntu       4096 Jul  7 06:57 /home/ubuntu/cephtest/archive
, 2023-07-07T06:57:45.506 INFO:teuthology.orchestra.run.smithi018.stdout:   393226      4 drwxr-xr-x   2  ubuntu   ubuntu       4096 Jul  7 06:00 /home/ubuntu/cephtest/archive/coredump

2023-07-07T17:34:06.527 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.0.0-4717-g6e5383fd (6e5383fdd6d402108195ddbbefb66fca2cebcf96) reef (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7fa928f2ced4]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x61b) [0x560d0fe70535]

2023-07-07T17:34:06.528 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:00:53.399 INFO:teuthology.orchestra.run.smithi018.stdout:
, 2023-07-07T06:00:53.400 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":27,"fsid":"c56c1d67-6b97-42b4-961e-90cf7e842363","created":"2023-07-07T05:59:50.010948+0000","modified":"2023-07-07T06:00:53.026768+0000","last_up_change":"2023-07-07T06:00:39.977315+0000","last_in_change":"2023-07-07T05:59:52.305625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-07T06:00:00.259409+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-07T06:00:00.646247+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":0.75,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2023-07-07T06:00:07.792277+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":1.5099999904632568,"score_stable":1.5099999904632568,"optimal_score":0.75,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b1a51f9-1515-4a38-b5dd-30e7dab92edd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6803","nonce":1830563861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":1830563861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6807","nonce":1830563861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":1830563861}]},"public_addr":"172.21.15.18:6803/1830563861","cluster_addr":"172.21.15.18:6802/1830563861","heartbeat_back_addr":"172.21.15.18:6807/1830563861","heartbeat_front_addr":"172.21.15.18:6806/1830563861","state":["exists"]},{"osd":1,"uuid":"e16c3d27-989a-43a5-b811-7a8029956383","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6801","nonce":45406582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":45406582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6805","nonce":45406582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":45406582}]},"public_addr":"172.21.15.18:6801/45406582","cluster_addr":"172.21.15.18:6800/45406582","heartbeat_back_addr":"172.21.15.18:6805/45406582","heartbeat_front_addr":"172.21.15.18:6804/45406582","state":["exists","up"]},{"osd":2,"uuid":"d8a24ad0-d28f-4794-bc1e-14191a09cafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2277875865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":2277875865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2277875865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":2277875865}]},"public_addr":"172.21.15.44:6804/2277875865","cluster_addr":"172.21.15.44:6805/2277875865","heartbeat_back_addr":"172.21.15.44:6808/2277875865","heartbeat_front_addr":"172.21.15.44:6809/2277875865","state":["exists","up"]},{"osd":3,"uuid":"fc7f5fb6-136c-406b-8f74-17ffe6776ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6803","nonce":3912049469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3912049469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3912049469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6807","nonce":3912049469}]},"public_addr":"172.21.15.44:6803/3912049469","cluster_addr":"172.21.15.44:6802/3912049469","heartbeat_back_addr":"172.21.15.44:6806/3912049469","heartbeat_front_addr":"172.21.15.44:6807/3912049469","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2023-07-07T06:00:39.977315+0000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2023-07-07T06:00:56.415 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json

2023-07-07T17:34:06.529 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:01:54.103 INFO:teuthology.orchestra.run.smithi018.stdout:
, 2023-07-07T06:01:54.103 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":27,"fsid":"c56c1d67-6b97-42b4-961e-90cf7e842363","created":"2023-07-07T05:59:50.010948+0000","modified":"2023-07-07T06:00:53.026768+0000","last_up_change":"2023-07-07T06:00:39.977315+0000","last_in_change":"2023-07-07T05:59:52.305625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-07T06:00:00.259409+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-07T06:00:00.646247+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":0.75,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2023-07-07T06:00:07.792277+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":1.5099999904632568,"score_stable":1.5099999904632568,"optimal_score":0.75,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b1a51f9-1515-4a38-b5dd-30e7dab92edd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6803","nonce":1830563861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":1830563861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6807","nonce":1830563861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":1830563861}]},"public_addr":"172.21.15.18:6803/1830563861","cluster_addr":"172.21.15.18:6802/1830563861","heartbeat_back_addr":"172.21.15.18:6807/1830563861","heartbeat_front_addr":"172.21.15.18:6806/1830563861","state":["exists"]},{"osd":1,"uuid":"e16c3d27-989a-43a5-b811-7a8029956383","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6801","nonce":45406582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":45406582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6805","nonce":45406582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":45406582}]},"public_addr":"172.21.15.18:6801/45406582","cluster_addr":"172.21.15.18:6800/45406582","heartbeat_back_addr":"172.21.15.18:6805/45406582","heartbeat_front_addr":"172.21.15.18:6804/45406582","state":["exists","up"]},{"osd":2,"uuid":"d8a24ad0-d28f-4794-bc1e-14191a09cafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2277875865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":2277875865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2277875865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":2277875865}]},"public_addr":"172.21.15.44:6804/2277875865","cluster_addr":"172.21.15.44:6805/2277875865","heartbeat_back_addr":"172.21.15.44:6808/2277875865","heartbeat_front_addr":"172.21.15.44:6809/2277875865","state":["exists","up"]},{"osd":3,"uuid":"fc7f5fb6-136c-406b-8f74-17ffe6776ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6803","nonce":3912049469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3912049469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3912049469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6807","nonce":3912049469}]},"public_addr":"172.21.15.44:6803/3912049469","cluster_addr":"172.21.15.44:6802/3912049469","heartbeat_back_addr":"172.21.15.44:6806/3912049469","heartbeat_front_addr":"172.21.15.44:6807/3912049469","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2023-07-07T06:00:39.977315+0000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2023-07-07T06:01:57.118 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json

2023-07-07T17:34:06.530 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:02:54.627 INFO:teuthology.orchestra.run.smithi018.stdout:
, 2023-07-07T06:02:54.627 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":27,"fsid":"c56c1d67-6b97-42b4-961e-90cf7e842363","created":"2023-07-07T05:59:50.010948+0000","modified":"2023-07-07T06:00:53.026768+0000","last_up_change":"2023-07-07T06:00:39.977315+0000","last_in_change":"2023-07-07T05:59:52.305625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-07T06:00:00.259409+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-07T06:00:00.646247+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":0.75,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2023-07-07T06:00:07.792277+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":1.5099999904632568,"score_stable":1.5099999904632568,"optimal_score":0.75,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b1a51f9-1515-4a38-b5dd-30e7dab92edd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6803","nonce":1830563861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":1830563861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6807","nonce":1830563861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":1830563861}]},"public_addr":"172.21.15.18:6803/1830563861","cluster_addr":"172.21.15.18:6802/1830563861","heartbeat_back_addr":"172.21.15.18:6807/1830563861","heartbeat_front_addr":"172.21.15.18:6806/1830563861","state":["exists"]},{"osd":1,"uuid":"e16c3d27-989a-43a5-b811-7a8029956383","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6801","nonce":45406582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":45406582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6805","nonce":45406582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":45406582}]},"public_addr":"172.21.15.18:6801/45406582","cluster_addr":"172.21.15.18:6800/45406582","heartbeat_back_addr":"172.21.15.18:6805/45406582","heartbeat_front_addr":"172.21.15.18:6804/45406582","state":["exists","up"]},{"osd":2,"uuid":"d8a24ad0-d28f-4794-bc1e-14191a09cafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2277875865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":2277875865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2277875865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":2277875865}]},"public_addr":"172.21.15.44:6804/2277875865","cluster_addr":"172.21.15.44:6805/2277875865","heartbeat_back_addr":"172.21.15.44:6808/2277875865","heartbeat_front_addr":"172.21.15.44:6809/2277875865","state":["exists","up"]},{"osd":3,"uuid":"fc7f5fb6-136c-406b-8f74-17ffe6776ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6803","nonce":3912049469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3912049469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3912049469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6807","nonce":3912049469}]},"public_addr":"172.21.15.44:6803/3912049469","cluster_addr":"172.21.15.44:6802/3912049469","heartbeat_back_addr":"172.21.15.44:6806/3912049469","heartbeat_front_addr":"172.21.15.44:6807/3912049469","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2023-07-07T06:00:39.977315+0000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2023-07-07T06:02:57.642 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json

2023-07-07T17:34:06.531 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:03:55.356 INFO:teuthology.orchestra.run.smithi018.stdout:
, 2023-07-07T06:03:55.356 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":27,"fsid":"c56c1d67-6b97-42b4-961e-90cf7e842363","created":"2023-07-07T05:59:50.010948+0000","modified":"2023-07-07T06:00:53.026768+0000","last_up_change":"2023-07-07T06:00:39.977315+0000","last_in_change":"2023-07-07T05:59:52.305625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-07T06:00:00.259409+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-07T06:00:00.646247+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":0.75,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2023-07-07T06:00:07.792277+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":1.5099999904632568,"score_stable":1.5099999904632568,"optimal_score":0.75,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b1a51f9-1515-4a38-b5dd-30e7dab92edd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6803","nonce":1830563861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":1830563861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6807","nonce":1830563861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":1830563861}]},"public_addr":"172.21.15.18:6803/1830563861","cluster_addr":"172.21.15.18:6802/1830563861","heartbeat_back_addr":"172.21.15.18:6807/1830563861","heartbeat_front_addr":"172.21.15.18:6806/1830563861","state":["exists"]},{"osd":1,"uuid":"e16c3d27-989a-43a5-b811-7a8029956383","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6801","nonce":45406582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":45406582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6805","nonce":45406582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":45406582}]},"public_addr":"172.21.15.18:6801/45406582","cluster_addr":"172.21.15.18:6800/45406582","heartbeat_back_addr":"172.21.15.18:6805/45406582","heartbeat_front_addr":"172.21.15.18:6804/45406582","state":["exists","up"]},{"osd":2,"uuid":"d8a24ad0-d28f-4794-bc1e-14191a09cafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2277875865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":2277875865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2277875865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":2277875865}]},"public_addr":"172.21.15.44:6804/2277875865","cluster_addr":"172.21.15.44:6805/2277875865","heartbeat_back_addr":"172.21.15.44:6808/2277875865","heartbeat_front_addr":"172.21.15.44:6809/2277875865","state":["exists","up"]},{"osd":3,"uuid":"fc7f5fb6-136c-406b-8f74-17ffe6776ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6803","nonce":3912049469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3912049469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3912049469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6807","nonce":3912049469}]},"public_addr":"172.21.15.44:6803/3912049469","cluster_addr":"172.21.15.44:6802/3912049469","heartbeat_back_addr":"172.21.15.44:6806/3912049469","heartbeat_front_addr":"172.21.15.44:6807/3912049469","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2023-07-07T06:00:39.977315+0000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2023-07-07T06:03:58.372 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json

2023-07-07T17:34:06.532 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:04:56.197 INFO:teuthology.orchestra.run.smithi018.stdout:
, 2023-07-07T06:04:56.197 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":27,"fsid":"c56c1d67-6b97-42b4-961e-90cf7e842363","created":"2023-07-07T05:59:50.010948+0000","modified":"2023-07-07T06:00:53.026768+0000","last_up_change":"2023-07-07T06:00:39.977315+0000","last_in_change":"2023-07-07T05:59:52.305625+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-07T06:00:00.259409+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-07T06:00:00.646247+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":0.75,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2023-07-07T06:00:07.792277+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":1.5099999904632568,"score_stable":1.5099999904632568,"optimal_score":0.75,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b1a51f9-1515-4a38-b5dd-30e7dab92edd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6803","nonce":1830563861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":1830563861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6807","nonce":1830563861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":1830563861}]},"public_addr":"172.21.15.18:6803/1830563861","cluster_addr":"172.21.15.18:6802/1830563861","heartbeat_back_addr":"172.21.15.18:6807/1830563861","heartbeat_front_addr":"172.21.15.18:6806/1830563861","state":["exists"]},{"osd":1,"uuid":"e16c3d27-989a-43a5-b811-7a8029956383","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6801","nonce":45406582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":45406582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6805","nonce":45406582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":45406582}]},"public_addr":"172.21.15.18:6801/45406582","cluster_addr":"172.21.15.18:6800/45406582","heartbeat_back_addr":"172.21.15.18:6805/45406582","heartbeat_front_addr":"172.21.15.18:6804/45406582","state":["exists","up"]},{"osd":2,"uuid":"d8a24ad0-d28f-4794-bc1e-14191a09cafd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6804","nonce":2277875865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6805","nonce":2277875865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6808","nonce":2277875865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6809","nonce":2277875865}]},"public_addr":"172.21.15.44:6804/2277875865","cluster_addr":"172.21.15.44:6805/2277875865","heartbeat_back_addr":"172.21.15.44:6808/2277875865","heartbeat_front_addr":"172.21.15.44:6809/2277875865","state":["exists","up"]},{"osd":3,"uuid":"fc7f5fb6-136c-406b-8f74-17ffe6776ed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6803","nonce":3912049469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6802","nonce":3912049469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6806","nonce":3912049469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.44:6807","nonce":3912049469}]},"public_addr":"172.21.15.44:6803/3912049469","cluster_addr":"172.21.15.44:6802/3912049469","heartbeat_back_addr":"172.21.15.44:6806/3912049469","heartbeat_front_addr":"172.21.15.44:6807/3912049469","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2023-07-07T06:00:39.977315+0000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}
, 2023-07-07T06:04:59.220 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json

2023-07-07T17:34:06.533 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status
,     raise CommandFailedError(
, teuthology.exceptions.CommandFailedError: Command failed on smithi018 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json'

2023-07-07T17:34:06.533 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:57:31.882 DEBUG:tasks.ceph.mgr.x:waiting for process to exit
, 2023-07-07T06:57:31.882 INFO:teuthology.orchestra.run:waiting for 300
, 2023-07-07T06:57:31.883 INFO:tasks.ceph.mgr.x:Stopped

2023-07-07T17:34:06.534 WARNING:scrape:Ignoring malparsed backtrace:  49.6% -- replaced with /var/log/ceph/ceph-client.admin.111618.log.gz
, /var/log/ceph/ceph-client.admin.112206.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114435.log
,  49.6% -- replaced with /var/log/ceph/ceph-client.admin.112206.log.gz

2023-07-07T17:34:06.534 WARNING:scrape:Ignoring malparsed backtrace:  50.0% -- replaced with /var/log/ceph/ceph-client.admin.109033.log.gz
, /var/log/ceph/ceph-client.admin.110421.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113692.log
,  49.6% -- replaced with /var/log/ceph/ceph-client.admin.110421.log.gz

2023-07-07T17:34:06.535 WARNING:scrape:Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.110715.log:	gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110519.log
,  47.6% -- replaced with /var/log/ceph/ceph-client.admin.110715.log.gz
, /var/log/ceph/ceph-client.admin.114582.log:	gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log

2023-07-07T17:34:06.535 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:57:34.156 DEBUG:teuthology.orchestra.run.smithi018:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart
, 2023-07-07T06:57:34.158 DEBUG:teuthology.orchestra.run.smithi044:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart
, Redirecting to /bin/systemctl restart rsyslog.service

2023-07-07T17:34:06.536 WARNING:scrape:Ignoring malparsed backtrace: 2023-07-07T06:57:45.506 INFO:teuthology.orchestra.run.smithi018.stdout:   393224      4 drwxr-xr-x   3  ubuntu   ubuntu       4096 Jul  7 06:57 /home/ubuntu/cephtest
, 2023-07-07T06:57:45.506 INFO:teuthology.orchestra.run.smithi018.stdout:   393225      4 drwxr-xr-x   5  ubuntu   ubuntu       4096 Jul  7 06:57 /home/ubuntu/cephtest/archive
, 2023-07-07T06:57:45.506 INFO:teuthology.orchestra.run.smithi018.stdout:   393226      4 drwxr-xr-x   2  ubuntu   ubuntu       4096 Jul  7 06:00 /home/ubuntu/cephtest/archive/coredump

2023-07-07T17:34:07.410 INFO:scrape:Found 6 distinct failure reasons
2023-07-07T17:34:07.411 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 45 --op write 22 --op delete 10 --op write_excl 22 --pool unique_pool_0'
2023-07-07T17:34:07.411 INFO:scrape:1 jobs: ['7328300']
2023-07-07T17:34:07.411 INFO:scrape:suites: ['clusters/fixed-2', 'crimson-rados/basic/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/readwrite}']
2023-07-07T17:34:07.412 INFO:scrape:
2023-07-07T17:34:07.412 INFO:scrape:Timeout 120 running ceph --cluster ceph pg dump --format=json
2023-07-07T17:34:07.412 INFO:scrape:10 jobs: ['7328286', '7328298', '7328296', '7328302', '7328305', '7328293', '7328291', '7328288', '7328303', '7328299']
2023-07-07T17:34:07.413 INFO:scrape:suites intersection: ['crimson_qa_overrides', 'deploy/ceph']
2023-07-07T17:34:07.413 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/fixed-2', 'clusters/{fixed-2}', 'crimson-rados/basic/{centos_latest', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'tasks/rados_api_tests}', 'tasks/rados_python}', 'thrashers/default', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}', 'workloads/radosbench-high-concurrency}', 'workloads/small-objects-localized}', 'workloads/small-objects}', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects-localized}', 'workloads/snaps-few-objects}', 'workloads/write_fadvise_dontneed}']
2023-07-07T17:34:07.413 INFO:scrape:
2023-07-07T17:34:07.414 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh
2023-07-07T17:34:07.414 INFO:scrape:2 jobs: ['7328287', '7328290']
2023-07-07T17:34:07.414 INFO:scrape:suites intersection: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph']
2023-07-07T17:34:07.414 INFO:scrape:suites union: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_api_tests_old_format}', 'tasks/rbd_api_tests}']
2023-07-07T17:34:07.415 INFO:scrape:
2023-07-07T17:34:07.415 INFO:scrape:Dead: b'2023-07-07T17:33:15.972 INFO:teuthology.orchestra.run.smithi092.stderr:2023-07-07T17:33:15.969+0000 7fc46cdef700 -1 WARNING: the following dangerous and experimental features are enabled: crimson'
2023-07-07T17:34:07.415 INFO:scrape:1 jobs: ['7328289']
2023-07-07T17:34:07.416 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/pool-snaps-few-objects}']
2023-07-07T17:34:07.416 INFO:scrape:
2023-07-07T17:34:07.416 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op setattr 25 --op rmattr 25 --op copy_from 0 --op write_excl 50 --pool unique_pool_0'
2023-07-07T17:34:07.417 INFO:scrape:1 jobs: ['7328295']
2023-07-07T17:34:07.417 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects-balanced}']
2023-07-07T17:34:07.417 INFO:scrape:
2023-07-07T17:34:07.417 INFO:scrape:Failure: reached maximum tries (501) after waiting for 3000 seconds
2023-07-07T17:34:07.418 INFO:scrape:1 jobs: ['7328292']
2023-07-07T17:34:07.418 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/radosbench}']
2023-07-07T17:34:07.418 INFO:scrape:
2023-07-07T17:34:07.419 INFO:teuthology.results:Sending results to yingxin.cheng@intel.com: Test Run: yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/
failed:  15
dead:    0
running: 1
waiting: 0
queued:  0
passed:  4


Fail
=================================================================
[7328286]  crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_api_tests}
-----------------------------------------------------------------
time:   00:24:59
info:   http://pulpito.ceph.com/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328286/
log:    http://qa-proxy.ceph.com/teuthology/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328286/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8b370675c0b842ea987aa9003f081d0b

    Command failed on smithi083 with status 124: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph pg dump --format=json'

[7328287]  crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_api_tests}
-----------------------------------------------------------------
time:   03:17:31
info:   http://pulpito.ceph.com/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328287/
log:    http://qa-proxy.ceph.com/teuthology/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328287/
sentry: https://sentry.ceph.com/organizations/ceph/?query=30ebdd7a4b254099b266bd88a768d033

    Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on
    smithi173 with status 124: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=6e5383fdd6d402108195ddbbefb66fca2cebcf96
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 CRIMSON_COMPAT=1
    adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout
    3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crims
    on_librbd.sh'

[7328288]  crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests}
-----------------------------------------------------------------
time:   01:12:15
info:   http://pulpito.ceph.com/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328288/
log:    http://qa-proxy.ceph.com/teuthology/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328288/
sentry: https://sentry.ceph.com/organizations/ceph/?query=96ad765792544533ac6a3a4e9034d361

    Command failed on smithi077 with status 124: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph pg dump --format=json'

[7328290]  crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format}
-----------------------------------------------------------------
time:   03:19:04
info:   http://pulpito.ceph.com/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328290/
log:    http://qa-proxy.ceph.com/teuthology/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328290/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6ef255d83724480f8d753bc1a205f36c

    Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on
    smithi016 with status 124: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=6e5383fdd6d402108195ddbbefb66fca2cebcf96
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh'

[7328291]  crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench-high-concurrency}
-----------------------------------------------------------------
time:   01:13:08
info:   http://pulpito.ceph.com/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328291/
log:    http://qa-proxy.ceph.com/teuthology/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328291/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6948be0492d849e2a925f1e0fd7ddf86

    Command failed on smithi022 with status 124: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph pg dump --format=json'

[7328292]  crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench}
-----------------------------------------------------------------
time:   02:00:35
info:   http://pulpito.ceph.com/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328292/
log:    http://qa-proxy.ceph.com/teuthology/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328292/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fe299a3f2eb240e4be58507c2b320e26

    reached maximum tries (501) after waiting for 3000 seconds

[7328293]  crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_python}
-----------------------------------------------------------------
time:   00:26:38
info:   http://pulpito.ceph.com/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328293/
log:    http://qa-proxy.ceph.com/teuthology/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328293/
sentry: https://sentry.ceph.com/organizations/ceph/?query=07a41f51e518460ebadc9b0a2b11acfb

    Command failed on smithi060 with status 124: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph pg dump --format=json'

[7328295]  crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-balanced}
-----------------------------------------------------------------
time:   01:16:01
info:   http://pulpito.ceph.com/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328295/
log:    http://qa-proxy.ceph.com/teuthology/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328295/

    Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads
    --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-
    stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100
    --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op
    rollback 0 --op setattr 25 --op rmattr 25 --op copy_from 0 --op write_excl
    50 --pool unique_pool_0'

[7328296]  crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-localized}
-----------------------------------------------------------------
time:   01:14:20
info:   http://pulpito.ceph.com/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328296/
log:    http://qa-proxy.ceph.com/teuthology/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328296/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f2181d1a51fa4cf781dc60a5b41915cd

    Command failed on smithi008 with status 124: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph pg dump --format=json'

[7328298]  crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects}
-----------------------------------------------------------------
time:   01:16:37
info:   http://pulpito.ceph.com/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328298/
log:    http://qa-proxy.ceph.com/teuthology/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328298/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c03db953a93a4e5e84962b01fd727373

    Command failed on smithi026 with status 124: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph pg dump --format=json'

[7328299]  crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-balanced}
-----------------------------------------------------------------
time:   01:13:19
info:   http://pulpito.ceph.com/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328299/
log:    http://qa-proxy.ceph.com/teuthology/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328299/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8efd57afd4eb4f24b04d8c55f79f2725

    Command failed on smithi158 with status 124: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph pg dump --format=json'

[7328300]  crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/readwrite}
-----------------------------------------------------------------
time:   00:22:38
info:   http://pulpito.ceph.com/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328300/
log:    http://qa-proxy.ceph.com/teuthology/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328300/

    Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000
    --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000
    --max-stride-size 800000 --max-seconds 0 --op read 45 --op write 22 --op
    delete 10 --op write_excl 22 --pool unique_pool_0'

[7328302]  crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-localized}
-----------------------------------------------------------------
time:   01:13:22
info:   http://pulpito.ceph.com/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328302/
log:    http://qa-proxy.ceph.com/teuthology/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328302/
sentry: https://sentry.ceph.com/organizations/ceph/?query=057243952dc44a568ea69f8b014d9bae

    Command failed on smithi143 with status 124: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph pg dump --format=json'

[7328303]  crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects}
-----------------------------------------------------------------
time:   01:13:28
info:   http://pulpito.ceph.com/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328303/
log:    http://qa-proxy.ceph.com/teuthology/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328303/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a0986b04d27047a38f25792e0894f37d

    Command failed on smithi069 with status 124: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph pg dump --format=json'

[7328305]  crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/write_fadvise_dontneed}
-----------------------------------------------------------------
time:   01:13:46
info:   http://pulpito.ceph.com/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328305/
log:    http://qa-proxy.ceph.com/teuthology/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328305/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1be9484e47e54d3eb830240261bd4ea3

    Command failed on smithi141 with status 124: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph pg dump --format=json'



Running
=================================================================
[7328289] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/pool-snaps-few-objects}
info:   http://pulpito.ceph.com/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328289/



Pass
=================================================================
[7328294] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests}
time:   00:25:14
info:   http://pulpito.ceph.com/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328294/

[7328297] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_lock_and_fence}
time:   00:18:00
info:   http://pulpito.ceph.com/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328297/

[7328301] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests}
time:   00:40:46
info:   http://pulpito.ceph.com/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328301/

[7328304] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests_old_format}
time:   00:31:53
info:   http://pulpito.ceph.com/yingxin-2023-07-07_05:31:46-crimson-rados-wip-yingxin-multi-core-enabled-with-fix-1-distro-default-smithi/7328304/